Associate Staff Data Engineer

Summary

Join our dynamic and global Digital Product Engineering company as a skilled Data Engineer! We are seeking an experienced professional to manage scheduled DAGs and data workflows using modern data engineering tools and technologies within the Azure ecosystem. The ideal candidate will have expertise in Apache Airflow, Azure Data Factory, Azure Data Lake Storage, Databricks, PySpark, and Python. You will design, develop, and maintain data pipelines, troubleshoot workflows, and collaborate with cross-functional teams. This role requires strong problem-solving skills and comfort working in virtual machine environments via terminal. We offer a non-hierarchical work culture and the opportunity to work with a talented team of experts.

Requirements

• Must have Skills: Azure Data Factory, Azure Pipeline Management (Strong), Python (Strong), Databricks, Airflow
• Proficiency in Apache Airflow for orchestrating workflows and managing DAGs
• Strong experience with Azure cloud services, including ADF and ADLS
• Hands-on experience with Databricks and PySpark for big data processing and analytics
• Advanced programming skills in Python, with a focus on data manipulation and automation
• Comfort with terminal operations and commands in VM environments
• Solid understanding of data engineering principles, including data modeling, ETL processes, and data pipeline optimization
• Strong problem-solving skills and the ability to work independently in a dynamic environment

Responsibilities

• Design, develop, and manage workflows using Apache Airflow, ensuring scheduled DAGs run efficiently and reliably
• Work with Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS) for data integration and storage solutions
• Develop and maintain data pipelines and ETL processes using Databricks, PySpark, and Python
• Monitor, troubleshoot, and optimize data workflows and pipelines to ensure high availability and performance
• Collaborate with cross-functional teams to understand data requirements and deliver scalable solutions
• Manage and operate within virtual machine environments using terminal commands for development and debugging

Preferred Qualifications

• Experience with other Azure services or similar cloud platforms is a plus
• Familiarity with DevOps practices and tools for CI/CD in data engineering is a plus
• Knowledge of database technologies such as SQL and NoSQL systems is a plus
• Understanding of data security and compliance standards is a plus