Data Engineer

Summary

Join Avalere Health as an Azure Data Engineer and design, implement, and manage data solutions on the Microsoft Azure cloud platform. You will develop and implement data integration processes, handle large volumes of data using big data technologies, and work with various data storage options. Responsibilities include creating data processing workflows and pipelines to support data analytics and machine learning. The role requires proficiency in SQL, Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake Storage. You will also be involved in data migration, utilizing Data Factory and various SQL databases. Avalere Health offers a comprehensive benefits package including a generous retirement plan, private medical insurance, employee wellbeing initiatives, and a compelling time-off policy.

Requirements

• Proficiency in SQL, SQL Server
• Understand parallel processing and data architecture patterns
• Azure Data Factory (ADF)
• Azure Synapse Analytics
• Azure Data Lake Storage
• Ability to design and implement pipelines in ADF
• Build complex efficient processes for transformation using SQL & ADF Pipelines
• Ensure secure, organised, and reliable data pipelines
• Troubleshoot all data quality issues
• Your core foundation will be data analytics which lead you to becoming a data engineer

Responsibilities

• Develop and implement data integration processes to extract data from diverse sources and load it into data warehouses or data lakes
• Ensure that data is effectively cleaned, transformed, and loaded
• Handle and analyze large volumes of data using big data technologies available on the Azure platform
• Create data processing workflows and pipelines to support data analytics, machine learning, and other data-driven applications
• Work with various data storage options provided by Microsoft Azure, including Azure SQL Database, Azure Data Lake Storage, and Azure Blob Storage
• Design data storage systems that scale, perform efficiently, and meet the organisation’s requirements
• Design and implement pipelines in ADF
• Build complex efficient processes for transformation using SQL & ADF Pipelines
• Ensure secure, organised, and reliable data pipelines
• Troubleshoot all data quality issues
• Use Data Factory, Dedicated and serverless SQL DB's and Syanpse
• Pull data from API's & other sources; stage and transform this data before converting to JSON and loading into the target using API endpoints
• Analyze all sources to understand the data (with business support)
• Build processes to validate and highlight exceptions that sit outside any migration rules
• Successfully load data into target on multiple passes and process any feedback into changes necessary for a successful go live

Preferred Qualifications

Use of Postman for testing outside of ADF

Benefits

• Generous company pension/retirement plan
• Private medical insurance
• Comprehensive employee wellbeing initiatives
• Compelling time-off policy
• Many other excellent employee benefits