Associate Staff Data Engineer

Summary

Join our dynamic and global Digital Product Engineering company as a Data Engineer! We're looking for an experienced professional to manage scheduled DAGs and data workflows using modern data engineering tools and technologies within the Azure ecosystem. You will design, develop, and manage workflows using Apache Airflow, work with Azure Data Factory and Azure Data Lake Storage, and develop data pipelines using Databricks, PySpark, and Python. The ideal candidate will have strong experience with Azure cloud services and proficiency in Apache Airflow. This role requires advanced programming skills in Python and comfort with terminal operations in VM environments. We offer a collaborative and challenging work environment.

Requirements

• Possess Must have Skills: Azure Data Factory, Azure Pipeline Management (Strong), Databricks, Airflow
• Demonstrate proficiency in Apache Airflow for orchestrating workflows and managing DAGs
• Have strong experience with Azure cloud services, including ADF and ADLS
• Possess hands-on experience with Databricks and PySpark for big data processing and analytics
• Have advanced programming skills in Python, with a focus on data manipulation and automation
• Be comfortable with terminal operations and commands in VM environments
• Have a solid understanding of data engineering principles, including data modeling, ETL processes, and data pipeline optimization
• Possess strong problem-solving skills and the ability to work independently in a dynamic environment

Responsibilities

• Design, develop, and manage workflows using Apache Airflow, ensuring scheduled DAGs run efficiently and reliably
• Work with Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS) for data integration and storage solutions
• Develop and maintain data pipelines and ETL processes using Databricks, PySpark, and Python
• Monitor, troubleshoot, and optimize data workflows and pipelines to ensure high availability and performance
• Collaborate with cross-functional teams to understand data requirements and deliver scalable solutions
• Manage and operate within virtual machine environments using terminal commands for development and debugging

Preferred Qualifications

• Have experience with other Azure services or similar cloud platforms
• Be familiar with DevOps practices and tools for CI/CD in data engineering
• Have knowledge of database technologies such as SQL and NoSQL systems
• Possess an understanding of data security and compliance standards