Big Data Engineer

Summary

Join our team as a Data Engineer and collaborate with cross-functional teams to deliver high-quality data solutions across various domains. You will be responsible for designing and implementing scalable data pipelines using Azure Databricks and Data Factory, ensuring data quality and integrity. This role requires expertise in Azure cloud technologies, data modeling, and programming languages like PySpark and Python. You will also provide technical support and stay current with industry best practices. The ideal candidate possesses 5+ years of IT experience and at least 4 years of experience with Azure Databricks.

Requirements

• 5+ years of IT experience
• Min. 4 years of Experience working with Azure Databricks
• Experience in Data Modelling & Source System Analysis
• Familiarity with PySpark
• Mastery of SQL
• Knowledge of components: Azure Data Factory, Azure Data Lake, Azure SQL DW, Azure SQL
• Experience with Python programming language used for data Engineering purposes
• Ability to conduct data profiling, cataloging, and mapping for technical design and construction of technical data flows
• Experience in data visualization/exploration tools
• Excellent communication skills, with the ability to effectively convey complex ideas to technical and non-technical stakeholders
• Strong team player with excellent interpersonal and collaboration skills

Responsibilities

• Be part of the technical plan for the migration, including data ingestion, transformation, storage, and access control in Azure's Data Factory and data lake
• Design and implement scalable and efficient data pipelines to ensure smooth data movement from multiple sources using Azure Databricks
• Developing scalable and re-usable frameworks for ingesting of data sets
• Ensure data quality and integrity throughout the entire data pipeline, implementing robust data validation and cleansing mechanisms
• Working with event based/streaming technologies to ingest and process data
• Provide support to the team, resolving any technical challenges or issues that may arise during the migration and post-migration phases
• Stay up to date with the latest advancements in cloud computing, data engineering, and analytics technologies, and recommend best practices and industry standards for implementing the data lake solution