The Lifetime Value Co. is hiring a
Web Crawling Engineer

Summary

Join Lifetime Value Co. as a Web Scraping Engineer on a long term contract to extract and ingest data from websites using web crawling tools, improve crawl/scrape analysis, reports and data management, test data and scrapes for accuracy, identify and rectify issues with breaks, scale scrapers as needed, and work closely with Data Scientists and Product Team to build future data pipelines.

Requirements

• Experience running large scale web scrapers; ideally some familiarity with a big data stack
• Experience with system monitoring/administration tools
• Experience with version control, open source practices, and code review
• Experience with applications designed to display archived web content
• Knowledge of entity resolution best practices and ontology creation
• Strong database creation and administration knowledge; MySQL and NoSQL (elastic, PostgresSQL, graph-dbs)
• Experience with streaming data sources and RESTful interfaces including familiarity with extracting data from publicly available API endpoints

Responsibilities

• Responsible for extracting and ingesting data from websites using web crawling tools
• Building and maintaining infrastructure to support those tools
• Own the creation process of these tools, services, and workflows to improve crawl/scrape analysis, reports and data management
• Test the data and the scrape to ensure accuracy and quality
• Own the process to identify and rectify any issues with breaks as well as scale scrapers as needed
• Building and managing targeted web scrapers, including but not limited to ad-hoc scraping tasks and production-level regularly recurring scraping jobs
• Managing the pipeline and storage for the data of those scrapers

Preferred Qualifications

• Experience in automotive industry web scraping a plus
• Experience extracting data from PDFs
• Experience in understanding Web Page Architecture
• Experience in digital image manipulation (converting images)
• Experience in AWS, RDS, Python, (especially Beautiful Soup) and Bright Data

Benefits

• Experience with Apache Airflow
• Experience working in an Agile development environment
• Experience programming in Python and/or Golang
• Experience with column storage (AWS Redshift, Google BigQuery)
• Experience with ETL tools
• Experience with public record data