Remote Senior Data Engineer

Summary

Join our team as a Data Engineer with 7+ years of experience and 5+ years of expertise in Azure technologies. You will participate in business discussions, gather data requirements, and utilize your strong analytical and problem-solving skills. Proficiency in SQL, PySpark, and Azure Data Factory is essential, along with experience in data pipeline creation and management. You will work with Databricks, Git, and collaborate effectively within an agile team environment. Excellent communication and documentation skills are required, along with a willingness to adapt to evolving technologies and project needs. Azure certification is mandatory.

Requirements

• Have 7+ years of overall experience
• Have more than 5+ years of expertise in Azure technologies
• Possess Azure certification
• Have knowledge of Azure Databricks, Data Lakehouse architectures, and Azure Data Factory
• Have expertise in Spark, PySpark, Delta Lake, Azure DevOps, and Python

Responsibilities

• Participate in business discussions and assist in gathering data requirements
• Utilize good analytical and problem-solving skills to address data challenges
• Write complex SQL queries for data extraction, transformation, and analysis
• Utilize knowledge of SQL functions, joins, subqueries, and performance tuning
• Navigate source systems with minimal guidance to understand how data is related and use data profiling to gain a better understanding of the data
• Use hands-on experience with PySQL/Pyspark
• Create and manage data pipelines using Azure Data Factory
• Demonstrate understanding of data integration, transformation, and workflow orchestration in Azure environments
• Understand data engineering workflows and best practices in Databricks
• Understand existing templates and patterns for development
• Use hands-on experience with Unity Catalog and Databricks workflow
• Use proficiency in Git for version control and collaboration in data projects
• Work effectively in a team environment, especially in agile or collaborative settings
• Communicate findings and recommendations clearly and effectively to other team members
• Document processes, workflows, and data analysis results effectively
• Show willingness to learn new tools, technologies, and techniques as the field of data analytics evolves
• Be adaptable to changing project requirements and priorities

Preferred Qualifications

• Have expertise in optimizing data workflows and predictive modeling
• Have experience in designing and implementing data pipelines using Databricks, Spark
• Have expertise in batch and streaming data solutions, automating workflows with CI/CD tools like Jenkins and Azure DevOps, and ensuring data governance with Delta Lake