Remote Senior GCP Data Engineer (Databricks)

Summary

Join Xebia as a Cloud Engineer and be responsible for designing, building, and deploying at-scale infrastructure with a focus on distributed systems. You will work closely with analysts/data scientists to understand impact to downstream data models and contribute to good software engineering practices across the team.

Requirements

• 3+ years’ experience with GCP (BigQuery, Dataflow, Pub/Sub, Bigtable or other NoSQL database, Dataproc, Storage, Kubernetes Engine
• 5+ years’ experience with data engineering or backend/fullstack software development
• Strong SQL skills
• Python scripting proficiency
• Experience with data transformation tools - Databricks and Spark
• Data manipulation libraries (such as Pandas, NumPy, PySpark)
• Experience in structuring and modelling data in both relational and non-relational forms
• Ability to elaborate and propose relational/non-relational approach
• Normalization / denormalization and data warehousing concepts (star, snowflake schemas)
• Designing for transactional and analytical operations
• Experience with CI/CD tooling (GitHub, Azure DevOps, Harness etc.)
• Good verbal and written communication skills in English
• Work from European Union region and work permit are required

Responsibilities

• Responsible for at-scale infrastructure design, build and deployment with a focus on distributed systems
• Building and maintaining architecture patterns for data processing, workflow definitions, and system to system integrations using Big Data and Cloud technologies
• Evaluating and translating technical design to workable technical solutions/code and technical specifications at par with industry standards
• Driving creation of re-usable artifacts
• Establishing scalable, efficient, automated processes for data analysis, data model development, validation, and implementation
• Working closely with analysts/data scientists to understand impact to the downstream data models
• Writing efficient and well-organized software to ship products in an iterative, continual release environment
• Contributing and promoting good software engineering practices across the team
• Communicating clearly and effectively to technical and non-technical audiences
• Defining data retention policies
• Monitoring performance and advising any necessary infrastructure changes