Senior Staff Big Data Engineer

Summary

Join our Digital Product Engineering company as a Highly Experienced Senior Developer Palantir and work with our Business Analytics & Insights (BAI) team. You will collaborate with cross-functional teams to deliver high-quality solutions across various domains. This role requires developing comprehensive technical plans for data migration, building scalable data ingestion frameworks, and ensuring data quality. You will provide technical guidance, stay updated on industry advancements, and translate business needs into data solutions. The position demands extensive experience with Palantir and cloud technologies, along with strong analytical, communication, and teamwork skills.

Requirements

• Possess Data Analysis & Visualization skills
• Have 8 to 10 years of IT experience
• Have Minimum 4 years of Experience working with Palantir
• Have Experience in Cloud (Azure or Google)
• Have Experience in Data Modelling & Source System Analysis
• Have Familiarity with PySpark
• Have Mastery of SQL
• Be able to conduct data profiling, cataloging, and mapping for technical design and construction of technical data flows
• Have Experience in data visualization/exploration tools
• Have Experience providing technical leadership and guidance to Teams
• Possess Strong problem-solving skills and the ability to translate business requirements into actionable data science solutions
• Possess Excellent communication skills, with the ability to effectively convey complex ideas to technical and non-technical stakeholders
• Be a Strong team player with excellent interpersonal and collaboration skills
• Be able to manage multiple projects simultaneously and deliver high-quality results within specified timelines

Responsibilities

• Develop a comprehensive technical plan for the migration, including data ingestion, transformation, storage, and access control in Azure's Data Factory and data lake
• Develop scalable and re-usable frameworks for ingesting of data sets
• Ensure data quality and integrity throughout the entire data pipeline, implementing robust data validation and cleansing mechanisms
• Work with event based / streaming technologies to ingest and process data
• Provide technical guidance and support to the team, resolving any technical challenges or issues that may arise during the migration and post-migration phases
• Stay up to date with the latest advancements in cloud computing, data engineering, and analytics technologies, and recommend best practices and industry standards for implementing the data lake solution

Preferred Qualifications

• Have Power BI skills (Capable)
• Have Experience preparing data for Data Science and Machine Learning Personal Attributes