Remote Senior Developer - Azure Databricks

Summary

Join us at Nagarro as a Senior Developer to work on high-quality solutions in various domains, utilizing your technical expertise to deliver scalable and efficient data pipelines.

Requirements

• 8 to 10 years of IT experience, with at least 4 years working with Azure Databricks
• Experience in data modeling and source system analysis
• Strong knowledge of PySpark, SQL, and data engineering using Python
• Proficiency with Azure Data Factory, Azure Data Lake, Azure SQL DW, and Azure SQL
• Ability to conduct data profiling, cataloging, and mapping for the design and construction of data flows
• Experience with data visualization and exploration tools
• Strong technical leadership and guidance skills
• Excellent problem-solving skills, with the ability to translate business requirements into data solutions
• Effective communicator, able to explain complex concepts to both technical and non-technical stakeholders
• Collaborative team player with strong interpersonal skills
• Ability to manage multiple projects and deliver quality results within deadlines

Responsibilities

• Develop a comprehensive technical plan for the migration, including data ingestion, transformation, storage, and access control in Azure's Data Factory and data lake
• Design and implement scalable and efficient data pipelines to ensure smooth data movement from multiple sources using Azure Databricks
• Developing scalable and re-usable frameworks for ingesting data sets
• Ensure data quality and integrity throughout the entire data pipeline, implementing robust data validation and cleansing mechanisms
• Working with event based / streaming technologies to ingest and process data
• Provide technical guidance and support to the team, resolving any technical challenges or issues that may arise during the migration and post-migration phases
• Stay up to date with the latest advancements in cloud computing, data engineering, and analytics technologies, and recommend best practices and industry standards for implementing the data lake solution

Preferred Qualifications

• Microsoft Certified: Azure Data Engineer Associate
• Experience preparing data for Data Science and Machine Learning
• Knowledge of Jupyter Notebooks or Databricks Notebooks for Python development