Senior Software Engineer, Data Ingestion

Summary

Join Astronomer's R&D team as a Software Engineer and contribute to the development of Astro Observe, our data observability product built on OpenLineage. You will play a key role in designing and implementing OpenLineage data collection and ingestion, enhancing the platform's usability, reliability, and scalability. Collaborate with cross-functional teams to understand user needs and iterate on observability and governance features. Contribute to the OpenLineage and Airflow Open Source communities. This role requires 5+ years of engineering experience, proficiency in Python, and strong communication skills. Bonus points for experience with OpenLineage, Airflow, data observability products, and other technologies.

Requirements

• A strong focus on helping customers gain more visibility and actionable insights through our observability offerings
• Proven experience with Python
• Strong written and verbal communication skills, with the ability to find middle ground
• 5+ years of engineering experience
• Experience with fast-paced startup environments
• Experience with software development best practices, such as code reviews, testing, CI/CD, version control, automation and debugging

Responsibilities

• Contribute to the OpenLineage and Airflow Open Source community to enhance the data collection and extraction of OpenLineage events from Airflow
• Contribute to the design, development and vision of Astro Observe’s ingestion pipelines
• Collaborate with cross-functional teams to understand user requirements, implement and iterate on the observability and governance features to help our customers gain visibility into their data pipelines and data products
• Contribute to the overall platform usability, reliability, and scalability

Preferred Qualifications

• Experience with OpenLineage
• Experience with Apache Airflow
• Experience with data observability products
• Experience with Golang
• Experience with SQL and OLAP databases
• Experience with distributed systems and stream processing

Benefits

• The estimated salary for this role ranges from CA$190,000 - CA$220,000 based on leveling and geography, along with an equity component and a comprehensive benefits package
• Remote work