Data Streaming Engineer

Summary

Join Xebia as a Data Engineer to develop and deliver high-quality features for a new Data Platform. You will independently create and maintain Flink jobs, build pipelines that consume events from Kafka into S3, and make data accessible on the Databricks platform. Collaborate with the Data Foundation Team to maintain the Data Platform and deliver with high autonomy. This role requires 6+ years of hands-on experience in Software Development/Data Engineering, advanced knowledge of Python or Scala, and experience with AWS, Flink, Kafka Connect, and CI/CD. A fundamental understanding of data file formats and experience with orchestrators are also necessary. Work from the European Union region with a valid work permit is required.

Requirements

• Ready to start within 4 weeks
• 6+ years of hands-on experience in Software Development/Data Engineering
• Advanced knowledge of Python and/or Scala
• Experience in AWS Cloud usage and management
• Experience with Flink and Kafka Connect
• Experience with CI/CD
• Fundamental understanding of Parquet, Delta Lake and other data file formats
• Experience with orchestrators such as Airflow or Kubeflow
• Familiarity with containerization and orchestration technologies (e.g., Docker, Kubernetes)
• Strong written and verbal English communication skills and proficient in communication with non-technical stakeholders
• Work from the European Union region and a work permit are required

Responsibilities

• Developing and delivering high-quality features: of a new data platform
• Creating and maintaining Flink jobs that send data to the event collector and set up dual writes with the current and target platforms
• Writing and managing pipelines that consume events from Kafka into S3 and making the data available in the Databricks platform
• Working closely with the Data Foundation Team to create and maintain the Data Platform
• Delivering with high autonomy and minimal guidance

Preferred Qualifications

• Experience with Databricks
• Experience with Strimzi
• Proficiency on an IaC tool such as Terraform, CDK or CloudFormation