Associate Staff Data Engineer

Summary

Join our dynamic and global Digital Product Engineering company as a Data Engineer! We are seeking an experienced professional to manage scheduled DAGs and data workflows using modern data engineering tools and technologies within the Azure ecosystem. The ideal candidate will have expertise in Apache Airflow, Azure Data Factory, Azure Data Lake Storage, Databricks, PySpark, and Python. You will design, develop, and manage workflows, ensuring efficient and reliable data pipelines. This role requires strong problem-solving skills and the ability to collaborate effectively with cross-functional teams. We offer a non-hierarchical work culture and a chance to work with a global team of experts.

Requirements

• Have Skills: Azure Data Factory, Azure Pipeline Management (Strong), Databricks, Airflow
• Proficiency in Apache Airflow for orchestrating workflows and managing DAGs
• Strong experience with Azure cloud services, including ADF and ADLS
• Hands-on experience with Databricks and PySpark for big data processing and analytics
• Advanced programming skills in Python, with a focus on data manipulation and automation
• Comfort with terminal operations and commands in VM environments
• Solid understanding of data engineering principles, including data modeling, ETL processes, and data pipeline optimization
• Strong problem-solving skills and the ability to work independently in a dynamic environment

Responsibilities

• Design, develop, and manage workflows using Apache Airflow, ensuring scheduled DAGs run efficiently and reliably
• Work with Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS) for data integration and storage solutions
• Develop and maintain data pipelines and ETL processes using Databricks, PySpark, and Python
• Monitor, troubleshoot, and optimize data workflows and pipelines to ensure high availability and performance
• Collaborate with cross-functional teams to understand data requirements and deliver scalable solutions
• Manage and operate within virtual machine environments using terminal commands for development and debugging

Preferred Qualifications

• Experience with other Azure services or similar cloud platforms
• Familiarity with DevOps practices and tools for CI/CD in data engineering
• Knowledge of database technologies such as SQL and NoSQL systems
• Understanding of data security and compliance standards