Lead Data Engineer

Summary

Join Integrate as a Lead Data Engineer and work with one of the world's largest cloud-based data lakes, leveraging your expertise in data warehouse solutions and large datasets. You will design and develop workflows and ETL processes, collaborate with cross-functional teams to define data needs, and utilize technologies like Scala, SQL, Snowflake, and BI tools. The role requires building data models, developing data architecture for reporting and AI/ML solutions, and creating reporting dashboards. You will also need to understand business use cases and develop meaningful analytic solutions. This position demands strong communication and leadership skills to mentor junior team members and stakeholders. Success in this role requires a passion for working with large datasets and a commitment to driving change through data-driven insights.

Requirements

• Advanced degree in Statistics, Computer Science or related technical/scientific field
• 9 + years experience in a Data Engineer development role
• Advanced knowledge of SQL, Python, and data processing workflow
• Strong experience and advanced technical skills writing APIs
• Extensive knowledge of Data Warehousing, ETL and BI architectures, concepts, and frameworks. And also strong in metadata definition, data migration and integration with emphasis on both high end OLTP and business Intelligence solutions
• Develop complex Stored procedure and queries to provide to the application along with reporting solutions too
• Optimize slow-running queries and optimize query performance
• Create optimized queries and data migration scripts
• Leadership skillsets to mentor and train junior team members and stakeholders
• Capable of creating long-term and short-term data architecture vision and tactical roadmap to achieve the data architecture vision beginning from the current state
• Strong data management abilities (i.e., understanding data reconciliations)
• Capable of facilitating data discovery sessions involving business subject matter experts
• Strong communication/partnership skills to gain the trust of stakeholders
• Knowledge of professional software engineering practices & best practices for the full software development lifecycle, including coding standards, code reviews, source control management, build processes, testing, and operations

Responsibilities

• Design and develop workflows, programs, and ETL to support data ingestion, curation, and provisioning of fragmented data for Data Analytics, Product Analytics and AI
• Work closely with Data Scientists, Software Engineers, Product Managers, Product Analysts and other key stakeholders to gather and define requirements for Integrate’s data needs
• Use Scala, SQL Snowflake, and BI tools to deliver data to customers
• Understand MongoDB/PostgreSQL and transactional data workflows
• Design data models and build data architecture that enables reporting, analytics, advanced AI/ML and Generative AI solutions
• Develop an understanding of the data and build business acumen
• Develop and maintain Datawarehouse and Datamart in the cloud using Snowflake
• Create reporting dashboards for internal and client stakeholders
• Understand the business use cases and customer value behind large sets of data and develop meaningful analytic solutions

Preferred Qualifications

• Industry experience as a Data Engineer or related specialty (e.g., Software Engineer, Business Intelligence Engineer, Data Scientist) with a track record of manipulating, processing, and extracting value from large datasets
• Experience building data products incrementally and integrating and managing datasets from multiple sources
• Query performance tuning skills using Unix profiling tools and SQL
• Experience leading large-scale data warehousing and analytics projects, including using AWS technologies – Snowflake, Redshift, S3, EC2, Data-pipeline and other big data technologies
• Nice to have Spark/Scala, MLFlow, and AWS experience