Data Engineer

Summary

Join Avalere Health as an Azure Data Engineer and design, implement, and manage data solutions on the Microsoft Azure cloud platform. You will develop and implement data integration processes, handle large volumes of data using big data technologies, and work with various data storage options. Responsibilities include creating data processing workflows and pipelines to support data analytics and machine learning. The role also involves data migration using Data Factory, SQL DBs, and Synapse, pulling data from APIs and other sources, and transforming data before loading it into the target. You will analyze data sources, build validation processes, and ensure successful data loading. This position requires proficiency in SQL, Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake Storage.

Requirements

• Proficiency in SQL, SQL Server
• Understand parallel processing and data architecture patterns
• Azure Data Factory (ADF)
• Azure Synapse Analytics
• Azure Data Lake Storage
• Ability to design and implement pipelines in ADF
• Build complex efficient processes for transformation using SQL & ADF Pipelines
• Ensure secure, organised, and reliable data pipelines
• Troubleshoot all data quality issues
• Your core foundation will be data analytics which lead you to becoming a data engineer

Responsibilities

• Develop and implement data integration processes to extract data from diverse sources and load it into data warehouses or data lakes
• Ensure that data is effectively cleaned, transformed, and loaded
• Handle and analyse large volumes of data using big data technologies available on the Azure platform
• Create data processing workflows and pipelines to support data analytics, machine learning, and other data-driven applications
• Build data processing pipelines, create analytics solutions, and enable data-driven decision-making within organizations using Azure cloud tools
• Work with various data storage options provided by Microsoft Azure, including Azure SQL Database, Azure Data Lake Storage, and Azure Blob Storage
• Design data storage systems that scale, perform efficiently, and meet the organisation’s requirements
• Use Data Factory, Dedicated and serverless SQL DB's and Syanpse
• Pull data from API's & other sources; stage and transform this data before converting to JSON and loading into the target using API endpoints
• Analyze all sources to understand the data (with business support)
• Build processes to validate and highlight exceptions that sit outside any migration rules
• Successfully load data into target on multiple passes and process any feedback into changes necessary for a successful go live

Preferred Qualifications

• Being able to onboard data from API's is essential
• Use of Postman for testing outside of ADF is a preferred skill

Benefits

• Up to a 7% pension contribution
• Life insurance
• Income protection
• Private medical insurance
• Flexible working arrangements, including flexible hybrid working
• Option to work from anywhere across the globe two weeks each year
• 25 days of annual leave plus two personal well-being days
• Gifted end-of-year holidays
• Early Summer Friday finish in June, July, and August
• Access free counselling through our employee assistance program
• Personalized health support
• Enhanced maternity, paternity, family leave, and fertility policies
• On-demand support from our partner Peppy
• Continuous opportunities to professionally develop with on-demand training, support, and global mobility opportunities across the business