Web Scraping Engineer

Summary

Join YipitData's growing Data Solutions team as a Web Scraping Engineer (Data Solutions Engineer). You will play a key role in designing, refactoring, and maintaining web scrapers crucial for company reports. Responsibilities include implementing advanced scraping techniques, collaborating with cross-functional teams, monitoring and troubleshooting scrapers, and driving continuous improvement. This fully remote position, based in India, offers flexibility in work hours. Success requires strong communication, extensive web scraping experience, and expertise in advanced fingerprinting and evasion strategies. YipitData offers a competitive salary and comprehensive benefits.

Requirements

• Effective communication in English with both technical and non-technical stakeholders
• 4+ years of experience with web scraping frameworks (e.g., Selenium, Playwright, or Puppeteer)
• Strong understanding of HTTP, RESTful APIs, HTML parsing, browser rendering, and TLS/SSL mechanics
• Expertise in advanced fingerprinting and evasion strategies (e.g., browser fingerprint spoofing, request signature manipulation)
• Deep experience managing cookies, headers, session states, and proxy rotations, including the deployment of both residential and data center proxies
• Experience with logging, metrics, and alerting to ensure high availability
• Troubleshooting skills to optimize scraper performance for efficiency, reliability, and scalability

Responsibilities

• Refactor and Maintain Web Scrapers
• Overhaul existing scraping scripts to improve reliability, maintainability, and efficiency
• Implement best coding practices (clean code, modular architecture, code reviews, etc.) to ensure quality and sustainability
• Implement Advanced Scraping Techniques
• Utilize sophisticated fingerprinting methods (cookies, headers, user-agent rotation, proxies) to avoid detection and blocking
• Handle dynamic content, navigate complex DOM structures, and manage session/cookie lifecycles effectively
• Collaborate with Cross-Functional Teams
• Work closely with analysts and other stakeholders to gather requirements, align on targets, and ensure data quality
• Support internal users of our web scraping tooling by providing troubleshooting, documentation, and best practices to ensure efficient data usage for critical reporting
• Monitor and Troubleshoot
• Develop robust monitoring solutions, alerting frameworks to quickly identify and address failures
• Continuously evaluate scraper performance, proactively diagnosing bottlenecks and scaling issues
• Drive Continuous Improvement
• Propose new tooling, methodologies, and technologies to enhance our scraping capabilities and processes
• Stay up to date with industry trends, evolving bot-detection tactics, and novel approaches to web data extraction

Benefits

• Vacation time
• Parental leave
• Team events
• Learning reimbursement
• Competitive salary