Remote Principal Data Engineer

Summary

The job is for a Principal Data Engineer (Azure) at Tiger Analytics, a global AI and analytics consulting firm with offices worldwide. The role involves working on Azure, Databricks, and data modeling to build and learn about various analytics solutions using different technologies. The employee will design and build scalable data pipelines, execute high-performance data processing, collaborate with multiple teams, and more.

Requirements

• Experience in implementing Data Lake with technologies like Azure Data Factory (ADF), PySpark, Databricks, ADLS, Azure SQL Database
• Strong programming, unit testing & debugging skills in SQL, Python or Scala/Java
• Experience in using big data technologies like Hadoop, Spark, Airflow, NiFi, Kafka, Hive, Neo4J, Elastic Search
• Good understanding of different file formats like Delta Lake, Avro, Parquet, JSON and CSV
• Experience in using Agile projects and following DevOps processes with technologies like Git, Jenkins & Azure DevOps

Responsibilities

• Design and build scalable & metadata-driven data ingestion pipelines (For Batch and Streaming Datasets)
• Conceptualize and execute high-performance data processing for structured and unstructured data, and data harmonization
• Schedule, orchestrate, and validate pipelines
• Design exception handling and log monitoring for debugging
• Ideate with your peers to make tech stack and tools-related decisions
• Interact and collaborate with multiple teams (Consulting/Data Science & App Dev) and various stakeholders to meet deadlines, to bring Analytical Solutions to life

Preferred Qualifications

• Comprehensive foundation with working knowledge of Azure Synapse Analytics, Event Hub & Streaming Analytics, Cosmos DB and Purview
• Experience of working on Data Lake & Lakehouse projects
• Experience of building REST services and implementing service-oriented architectures
• Experience of supporting BI and Data Science teams in consuming the data in a secure and governed manner
• Certifications like Data Engineering on Microsoft Azure (DP-203) or Databricks Certified Developer (DE)

Benefits

Compensation packages are among the best in the industry