COMPLY is hiring a
Web Scraping Data Engineering Manager in United States

Summary

The Data Engineering Manager at COMPLY will lead the design and implementation of advanced web scraping tools and services, ensuring high accuracy and quality. They will also mentor staff and manage offshore contractors, refactor web scraping frameworks, and collaborate with cross-functional teams to align activities with strategic goals.

Requirements

• Bachelor’s or higher in Computer science, engineering, analytics, or information technology or similar
• 10+ years in data engineering with 2+ years focused on web scraping
• 3+ years managing a team
• High Proficiency in Python
• Experience with Celery, RabbitMQ, and Django to manage existing web scraping processes
• Understanding of Elasticsearch indices, complex queries, and search optimization
• Deep knowledge in Scrapy, Beautiful Soup, Selenium or similar framework
• Thorough understanding of HTML, CSS, and JavaScript

Responsibilities

• Develop and refine advanced web scraping tools and services to efficiently extract and ingest data from diverse web sources
• Implement comprehensive monitoring systems to track the performance and health of web scraping operations
• Conduct rigorous testing to ensure the accuracy and reliability of data collected through web scraping
• Lead a team of geographically distributed data engineers, providing mentorship and direction
• Quickly identify and resolve technical issues and discrepancies in the data collection process
• Continuously evaluate and improve scraping processes and methodologies
• Collaborate with cross-functional teams to align web scraping activities with overall business objectives and strategic goals
• Ensure timely execution of project deliverables by overseeing project schedules and resource allocation
• Proactively stay current with advancements and trends in web scraping technologies

Preferred Qualifications

• Experience in financial services, political contributions or other relevant industries
• Certifications. : AWS (preferred), Azure, or GCP. Data specific certification a plus
• Other Technologies: Kubernetes, Redis, Jenkins, Postgres, ETL, dbt, NoSQL, SQL

Benefits

$120,000 - $140,000 a year