Data Engineer

Summary

Join Encora as a Data Engineer specializing in Apache Airflow to contribute to a large-scale data modernization project. You will design, implement, and manage data pipelines, migrating legacy SQL Server Agent jobs to Airflow. This role requires collaboration with internal client data engineers and focuses on automating critical data processes related to real estate Deeds data. The position demands strong Python and SQL skills, experience with AWS services, and expertise in data pipeline orchestration. The ideal candidate will have a proven track record of migrating legacy systems to Airflow and possess excellent problem-solving and communication skills. Work is fully remote, and the position is full-time.

Requirements

• 5+ years of hands-on experience developing and managing data pipelines using Apache Airflow in a production environment
• Proven experience migrating legacy orchestration systems, such as SQL Server Agent jobs, to Apache Airflow
• Strong proficiency in Python and SQL, with a deep understanding of data structures, algorithms, and best practices for writing efficient and maintainable code
• Familiarity with AWS cloud services relevant to data processing, including S3, EMR, Glue, and Kinesis
• Experience working with dbt for data transformation and modeling
• Excellent problem-solving and debugging skills, with the ability to identify and resolve complex data pipeline issues effectively
• Strong communication and collaboration skills, with the ability to work effectively in a team environment and interact with technical and non-technical stakeholders

Responsibilities

• Design, develop, and maintain scalable and efficient data pipelines using Apache Airflow to orchestrate a wide range of data processes, including data ingestion, transformation, validation, and loading
• Specifically focus on migrating existing SQL Server Agent jobs related to Deeds and parcel data processes to Apache Airflow
• Collaborate with the client’s internal data engineers, who will be assigned part-time to guide and support the Airflow implementation
• Take ownership of automating critical data processes currently managed by 1 client FTE, ensuring a smooth transition and knowledge transfer
• Seamlessly integrate Apache Airflow workflows with existing AWS Glue and dbt processes to create a unified and cohesive data pipeline orchestration system
• Implement robust monitoring, logging, and alerting mechanisms for Airflow pipelines to ensure data quality, identify potential issues, and facilitate proactive problem resolution
• Contribute to the development and maintenance of comprehensive documentation for all Airflow pipelines, ensuring clarity and ease of maintenance for the team
• Work closely with the Solution Architect, Data Architect, Data Migration Specialist, Cloud Engineer, and other team members to ensure seamless integration of Airflow pipelines within the broader data platform modernization project
• Actively participate in technical discussions and decision-making processes, providing insights and expertise on Airflow best practices and implementation strategies
• Communicate effectively with stakeholders, providing clear and concise updates on the progress of Airflow development and implementation, addressing any concerns, and ensuring alignment with project goals

Preferred Qualifications

• Experience with cloud-native solutions on AWS, including AWS Aurora and Amazon S3
• Familiarity with data governance and security best practices
• Experience with DevOps practices and CI/CD pipelines
• Contributions to the Apache Airflow open-source community

Benefits

Work from home