Data Engineer

Summary

Join YipitData's growing engineering team as a Web Scraping Engineer. You will design, build, and maintain robust web scrapers crucial for reports and customer experiences. This hands-on role involves tackling complex scraping challenges, collaborating with cross-functional teams, and ensuring data ingestion processes are efficient and scalable. You will refactor and maintain existing scrapers, implement advanced scraping techniques, and collaborate with analysts and stakeholders. The position also includes monitoring, troubleshooting, and driving continuous improvement in scraping capabilities. This is a fully remote opportunity based in India with flexible work hours.

Requirements

• Effective communication in English with both technical and non-technical stakeholders
• You have a track record of mentoring engineers and managing performance in a fast-paced environment
• 3+ years of experience with web scraping frameworks (e.g., Selenium, Playwright, or Puppeteer )
• Strong understanding of HTTP, RESTful APIs, HTML parsing, browser rendering, and TLS/SSL mechanics
• Expertise in advanced fingerprinting and evasion strategies (e.g., browser fingerprint spoofing, request signature manipulation)
• Deep experience managing cookies, headers, session states, and proxy rotations , including the deployment of both residential and data center proxies
• Experience with logging, metrics, and alerting to ensure high availability
• Troubleshooting skills to optimize scraper performance for efficiency, reliability, and scalability

Responsibilities

• Refactor and Maintain Web Scrapers
• Overhaul existing scraping scripts to improve reliability, maintainability, and efficiency
• Implement best coding practices (clean code, modular architecture, code reviews, etc.) to ensure quality and sustainability
• Implement Advanced Scraping Techniques
• Utilize sophisticated fingerprinting methods (cookies, headers, user-agent rotation, proxies) to avoid detection and blocking
• Handle dynamic content, navigate complex DOM structures, and manage session/cookie lifecycles effectively
• Collaborate with Cross-Functional Teams
• Work closely with analysts and other stakeholders to gather requirements, align on targets, and ensure data quality
• Provide support, documentation, and best practices to internal stakeholders to ensure effective use of our web scraped data in critical reporting workflows
• Monitor and Troubleshoot
• Develop robust monitoring solutions, alerting frameworks to quickly identify and address failures
• Continuously evaluate scraper performance, proactively diagnosing bottlenecks and scaling issues
• Drive Continuous Improvement
• Propose new tooling, methodologies, and technologies to enhance our scraping capabilities and processes
• Stay up to date with industry trends, evolving bot-detection tactics, and novel approaches to web data extraction

Benefits

• Flexible work hours
• Flexible vacation
• A generous 401K match
• Parental leave
• Team events
• Wellness budget
• Learning reimbursement