Senior Azure Data Engineer

Summary

Join our team as a Data Engineer and contribute to business discussions by gathering data requirements. You will leverage your analytical and problem-solving skills to address data challenges, and your proficiency in writing complex SQL queries for data extraction, transformation, and analysis will be essential. Experience with Azure Data Factory, Databricks, and Git is required. You will design and implement data pipelines, optimize data workflows, and work collaboratively in a team environment. Excellent communication and documentation skills are necessary to articulate findings and recommendations. A willingness to learn new technologies and adapt to changing project requirements is also crucial.

Requirements

• Proficiency in Azure Databricks, Data Lakehouse architectures, and Azure Data Factory is required
• Expertise in optimizing data workflows and predictive modeling is required
• Experience in designing and implementing data pipelines using Databricks and Spark is required
• Expertise in batch and streaming data solutions is required
• Experience in automating workflows with CI/CD tools like Jenkins and Azure DevOps is required
• Experience in ensuring data governance with Delta Lake is required
• Proficiency in Spark, PySpark, Delta Lake, Azure DevOps, and Python is required
• Advanced SQL expertise is required

Responsibilities

• Participate in business discussions and assist in gathering data requirements
• Employ good analytical and problem-solving skills to address data challenges
• Write complex SQL queries for data extraction, transformation, and analysis
• Utilize SQL functions, joins, subqueries, and perform performance tuning
• Navigate source systems with minimal guidance to understand data relationships and use data profiling to gain a better understanding of the data
• Have hands-on experience with PySQL/Pyspark
• Create and manage data pipelines using Azure Data Factory
• Understand data integration, transformation, and workflow orchestration in Azure environments
• Understand data engineering workflows and best practices in Databricks
• Understand existing templates and patterns for development
• Have hands-on experience with Unity Catalog and Databricks workflow
• Use Git for version control and collaboration in data projects
• Work effectively in a team environment, especially in agile or collaborative settings
• Communicate findings and recommendations clearly and effectively to other team members
• Document processes, workflows, and data analysis results effectively
• Be willing to learn new tools, technologies, and techniques as the field of data analytics evolves
• Be adaptable to changing project requirements and priorities