Senior Data Engineer

Summary

Join MediaRadar as a Senior Data Engineer and take ownership of our data infrastructure, architecting, building, and scaling robust data pipelines. You will lead and mentor a team of data engineers, ensuring data integrity through comprehensive testing and validation. Responsibilities include optimizing system performance, troubleshooting production issues, and collaborating with cross-functional teams. The ideal candidate possesses a bachelor's or master's degree in a related field and 7+ years of hands-on experience in data engineering with expertise in Databricks, Apache Spark, and Python. This remote position requires availability during core working hours in the EST/CST time zones.

Requirements

• A bachelor’s or master’s degree in computer science, Engineering, or a related field (or equivalent practical experience)
• 7+ years of hands-on experience in data engineering, with a strong focus on Databricks, Apache Spark and 1 RDBMS solution for processing large-scale datasets
• Expert-level proficiency in Python (including PySpark) and a strong understanding of data structures, algorithms, and OOP principles
• Deep expertise in SQL and Spark SQL, with proven experience writing and optimizing complex analytical queries
• Hands-on experience with the Databricks ecosystem, including Delta Lake, Unity Catalog, and Data frame APIs

Responsibilities

• Architect & Build: Design, implement, and optimize scalable and reliable ELT/ETL pipelines using Databricks, Spark, Python and Store Procedures. You will model complex data and build solutions for both batch and streaming workloads
• Lead & Mentor: Guide and support a team of data engineers, providing technical direction and resolving challenges. You will elevate the team's skills through mentorship and constructive code reviews to ensure we ship high-quality, efficient code
• Ensure Data Integrity: Develop and implement comprehensive testing frameworks, data validation rules, and QA plans to guarantee the accuracy and integrity of our data assets
• Optimize & Troubleshoot: Proactively monitor system performance, tune complex SQL queries, and troubleshoot production issues. You will perform root-cause analysis and implement lasting solutions to improve system health and reliability
• Collaborate & Innovate: Actively participate in an Agile environment (Scrum, sprints, backlog grooming) and collaborate with cross-functional teams. You'll help evaluate and introduce new technologies and best practices to continuously improve our data platform

Preferred Qualifications

• Experience building and maintaining CI/CD pipelines using tools like Azure DevOps
• Previous experience migrating a legacy data system to a modern, unified data platform
• Familiarity with containerization (Docker, Kubernetes)
• Experience working in a fast-paced, product-driven environment

Benefits

• Medical, Dental & Vision Insurance
• 401k with Company Match
• Flexible PTO
• Commuter Benefits
• Gym Discounts
• Summer Fridays