Senior Big Data Engineer

Summary

Join Nagarro, a digital product engineering company, as a Data Engineer and contribute to the development of high-quality solutions in various domains. You will be responsible for designing and implementing data pipelines, ensuring data quality, and staying up-to-date with the latest technologies. This role involves working closely with cross-functional teams and collaborating with stakeholders.

Requirements

• 5+ years of working experience in a role of Data Engineer
• Minimum 2 years of hands-on experience working with Azure Databricks
• Experience in Data Modelling & Source System Analysis
• Familiarity with PySpark
• Mastery of SQL
• Knowledge of components: Azure Data Factory, Azure Data Lake, Azure SQL DW, Azure SQL
• Experience with Python programming language used for data Engineering purposes
• Ability to conduct data profiling, cataloging, and mapping for technical design and construction of technical data flows
• Experience in data visualization/exploration tools
• Excellent communication skills, with the ability to effectively convey complex ideas to technical and non-technical stakeholders
• Strong team player with excellent interpersonal and collaboration skills

Responsibilities

• Be part of the technical plan for the migration, including data ingestion, transformation, storage, and access control in Azure's Data Factory and data lake
• Design and implement scalable and efficient data pipelines to ensure smooth data movement from multiple sources using Azure Databricks
• Developing scalable and reusable frameworks for ingesting data sets
• Ensure data quality and integrity throughout the entire data pipeline, implementing robust data validation and cleansing mechanisms
• Working with event-based/streaming technologies to ingest and process data
• Provide support to the team, resolving any technical challenges or issues that may arise during the migration and post-migration phases
• Stay up to date with the latest advancements in cloud computing, data engineering, and analytics technologies, and recommend best practices and industry standards for implementing the data lake solution