Principal Data Engineer

Summary

Join InMarket as a Principal Data Engineer and build the essential data infrastructure for our leading consumer intelligence platform. You will design and implement ETL pipelines, advocate for best practices, and architect complex cross-team projects. You'll mentor other engineers, communicate effectively with stakeholders, and identify areas for improvement. This role requires strong SQL, Python, and Spark skills, along with experience in data warehousing and Big Data technologies. We offer a competitive salary, stock options, flexible vacation, comprehensive health benefits, 401k matching, and unlimited PTO.

Requirements

• Strong SQL experience
• Expert in at least one data pipelining orchestration framework (Airflow, Luigi, etc.)
• Experience building ETL pipelines with Python, SQL, and Spark
• Strong software engineering skills in an Object Oriented language such as Java or Python
• Experience building and optimizing a data warehouse on a major cloud platform (Big Query preferred but not required)
• Experience and deep understanding of Big Data Technologies such as Hadoop and Spark
• Experience with building and optimizing large scale and high-performance systems
• Extensive knowledge of data related tools and architecture concepts in a major cloud platform (ideally GCP)
• Knowledge of Kubernetes, Docker, Airflow, Git, and CD/CI best practices
• Strong collaboration and communication skills within and across teams
• B.S. or M.S. in Computer Science or a related technical field
• At least 5 years of data engineering experience

Responsibilities

• Design and implement ETL pipelines in Apache Airflow, Big Query, Python, and Spark to transform various upstream sources data into curated data assets for use across InMarket’s vast array of data driven products
• Stay up to date with and advocate internally for Data Engineering best practices, technologies, and testing frameworks that will help shape InMarket’s technical future
• Architect and plan complex cross team projects involving highly distributed and high performance data platform systems
• Provide technical guidance to other engineers in order to help promote the growth of their knowledge and technical capabilities
• Communicate complex concepts and the results of the analyses in a clear and effective manner to Product and Engineering Managers
• Identify areas for improvement in existing pipelines and processes
• Understand business requirements and convert them to technical solutions

Preferred Qualifications

• Experience in Scala
• Experience in Apache Airflow
• Experience in optimizing Big Query performance
• Experience building a data governance platform in a regulated environment
• Experience with various data testing frameworks and technologies
• Familiarity with the digital media / advertising industry

Benefits

• Competitive salary, stock options, flexible vacation
• Medical, dental and Flexible Spending Account (FSA)
• Company Matched 401(k)
• Unlimited PTO (within reason)
• Talented co-workers and management
• Agile Development Program (For continued learning/professional development)
• Paid Paternity & Maternity Leave