Lead Data Pipeline Engineer

Summary

Join Two Six Technologies and leverage your engineering expertise and experience with machine learning (ML) and large language models (LLMs) to help decision-makers understand complex information spaces. As a Lead Data Pipeline Engineer, you will develop and enhance high-volume data pipelines, integrate data science models into applications, and build user-facing applications utilizing generative AI and LLMs. You will collaborate with product engineering teams, data scientists, and DevOps to integrate data science into our products. This role requires experience with server-side Python, AWS, and Agile development, and involves managing a small team of data pipeline engineers. The ideal candidate is passionate about applying technology to solve complex problems and deriving insights from high-volume data. This is a fully remote position.

Requirements

• Experience building scalable server-side applications and data pipelines using Python
• Experience working with ML, LLMs, and collaborating closely with data science teams
• Experience with Elastic or similar data stores, as well as with SQL database technologies, ideally Postgres
• Experience with AWS or similar cloud-based infrastructure
• Experience with the design, development, testing, and support of scalable, data-driven applications deployed to production SaaS environments
• Experience working independently and as a part of an Agile team

Responsibilities

• Developing and enhancing high-volume data pipelines where data is enriched using ML and NLP functionality, and then used to feed additional machine learning and data science
• Integrating models that data science creates into applications that are fast and scalable
• Building and enhancing user-facing applications that use generative AI and LLMs to help our customers gain insights from data faster
• Operating in a collaborative, agile environment, with a focus on taking action and working closely with peers to enable team success
• Creating proofs-of-concept and prototypes to quickly test ideas, as well as designing and building scalable, production-ready solutions
• Considering security best practices at every phase of software development
• Continuously learning and improving
• Working in a fully remote team with a diverse set of skills and experiences
• Independently identifying and solving problems, and questioning assumptions in pursuit of the right solutions
• Actively participating in the peer code review process
• Creating and executing both manual and automated testing
• Managing a small team of data pipeline engineers

Preferred Qualifications

• Prior experience as a platform engineer or ML Ops engineer
• Experience with processing, storaging, and querying of data in multiple natural languages
• Elastic certification
• AWS certification

Benefits

• Medical, dental, and vision insurance
• Life and disability insurance
• Retirement benefits
• Paid leave
• Tuition assistance and professional development