Senior Azure Data Engineer Team Lead

Summary

Join our team as a Data Engineer and contribute to business discussions by gathering data requirements. You will leverage your strong analytical and problem-solving skills to overcome data challenges. Proficiency in SQL, PySpark, and Azure Data Factory is essential for data extraction, transformation, and pipeline management. Experience with Databricks, including Unity Catalog and workflow management, is also required. You will utilize Git for version control and collaborate effectively within an agile team environment. Excellent communication and documentation skills are necessary to articulate findings and processes.

Requirements

• Proficiency in Azure Databricks, Data Lakehouse architectures, and Azure Data Factory
• Expertise in optimizing data workflows and predictive modeling
• Ability to design and implement data pipelines using Databricks, Spark
• Expertise in batch and streaming data solutions
• Ability to automate workflows with CI/CD tools like Jenkins and Azure DevOps
• Ability to ensure data governance with Delta Lake
• Proficiency in Spark, PySpark, Delta Lake, Azure DevOps, and Python
• Advanced SQL expertise

Responsibilities

• Participate in business discussions and assist gathering data requirements
• Employ good analytical and problem-solving skills to address data challenges
• Write complex SQL queries for data extraction, transformation, and analysis
• Utilize SQL functions, joins, subqueries, and perform performance tuning
• Navigate source systems to understand data relationships and use data profiling to gain a better understanding of the data
• Use PySQL/Pyspark for data manipulation
• Create and manage data pipelines using Azure Data Factory
• Demonstrate understanding of data integration, transformation, and workflow orchestration in Azure environments
• Apply knowledge of data engineering workflows and best practices in Databricks
• Understand existing templates and patterns for development
• Use Unity Catalog and Databricks workflow
• Use Git for version control and collaboration in data projects
• Work effectively in a team environment, especially in agile or collaborative settings
• Communicate findings and recommendations clearly and effectively to other team members
• Document processes, workflows, and data analysis results effectively
• Show willingness to learn new tools, technologies, and techniques as the field of data analytics evolves
• Be adaptable to changing project requirements and priorities