Remote Senior Data Engineer

Summary

Join our team as a Data Engineer to help address data challenges, create and manage data pipelines, and optimize data workflows.

Requirements

• 6+ years of overall experience with more than 5years of expertise in Azure technologies
• Expertise in optimizing data workflows and predictive modeling
• Designing and implementing data pipelines using Databricks, Spark
• Expertise in batch and streaming data solutions, automating workflows with CI/CD tools like Jenkins and Azure DevOps, and ensuring data governance with Delta Lake
• Spark, PySpark, Delta Lake, Azure DevOps, Python

Responsibilities

• Participate in business discussions
• Assist gathering data requirements
• Use good analytical and problem-solving skills to help address data challenges
• Write complex SQL queries for data extraction, transformation, and analysis
• Navigate source systems with minimal guidance to understand how data is related and use like data profiling to gain a better understanding of the data
• Create and manage data pipelines using Azure Data Factory
• Understand existing templates and patterns for development in Databricks
• Use Git for version control and collaboration in data projects
• Work effectively in a team environment, especially in agile or collaborative settings
• Articulate findings and recommendations for other team members
• Document processes, workflows, and data analysis results effectively