Big Data Engineer

Summary

Join Nagarro's Business Analytics & Insights (BAI) team as a Data Engineer and collaborate with cross-functional teams to deliver high-quality data solutions. You will lead technical planning for data migration within Azure, design and implement scalable data pipelines using Azure Databricks, develop reusable frameworks for large datasets, and ensure data quality. Responsibilities include working with event-based technologies, providing technical support, and staying current with advancements in cloud computing. This role requires 3+ years of IT experience and 4+ years with Azure Databricks, along with proficiency in data modeling, PySpark, SQL, and various Azure components. Strong problem-solving, communication, and teamwork skills are essential.

Requirements

• 3+ years of IT experience
• Minimum of 4 years working with Azure Databricks
• Proficiency in Data Modeling and Source System Analysis
• Strong knowledge of PySpark and SQL
• Experience with Azure components: Data Factory, Data Lake, SQL Data Warehouse (DW), and Azure SQL
• Experience with Python for data engineering purposes
• Ability to conduct data profiling, cataloging, and mapping for technical design and construction of data flows
• Familiarity with data visualization/exploration tools

Responsibilities

• Lead the technical planning for data migration, including data ingestion, transformation, storage, and access control within Azure Data Factory and Azure Data Lake
• Design and implement scalable, efficient data pipelines to ensure smooth data movement from multiple sources using Azure Databricks
• Develop reusable frameworks for the ingestion of large datasets
• Ensure data quality and integrity by implementing robust validation and cleansing mechanisms throughout the data pipeline
• Work with event-based/streaming technologies to ingest and process data in real-time
• Provide technical support to the team, resolving challenges during the migration and post-migration phases
• Stay current with the latest advancements in cloud computing, data engineering, and analytics technologies; recommend best practices and industry standards for data lake solutions

Preferred Qualifications

• Strong problem-solving capabilities, able to translate business requirements into data science solutions
• Excellent communication skills, with the ability to explain complex concepts to both technical and non-technical stakeholders
• Strong team player with excellent interpersonal and collaboration skills
• Ability to deliver high-quality results within specified timelines