Pathway is hiring a
Data Engineer

Summary

The job is for a Data Processing or Data Engineering role at Pathway, a deeptech start-up specializing in an ultra-performant Data Processing Framework. The role involves implementing data flows, setting up CDC interfaces, designing ETL pipelines, and contributing to benchmark framework design. The ideal candidate has experience with distributed data processing frameworks, streaming dataflows, message queues, message brokers, data schema, Kubernetes, deployments in Azure and AWS clouds, Python, SQL, and working for an innovative tech company.

Requirements

• Inside-out understanding of at least one major distributed data processing framework (Spark, Dask, Ray,...)
• 6 months+ experience working with a streaming dataflow framework (e.g.: Flink, Kafka Streams or ksqldb, Spark in streaming mode, Beam/Dataflow)
• Ability to set up distributed dataflows independently
• Experience with data streams: message queues, message brokers (Kafka), CDC
• Working familiarity with data schema and schema versioning concepts; Avro, Protobuf, or others
• Familiarities with Kubernetes
• Familiarity with deployments in both Azure and AWS clouds
• Good working knowledge of Python
• Good working knowledge of SQL
• Experienced in working for an innovative tech company (SaaS, IT infrastructure or similar preferred)

Responsibilities

• Implement the flow of data from client's warehouses up to Pathway's ingress
• Set up CDC interfaces for change streams between client data stores and i/o data processed by Pathway; ensuring data persistence for Pathway outputs
• Design ETL pipelines within Pathway
• Contribute to benchmark framework design (throughput / latency / memory footprint; consistency), including in a distributed system setup
• Contribute to building open-source test frameworks for simulated streaming data scenarios on public datasets

Preferred Qualifications

Warmly disposed towards open-source and open-core software, but pragmatic about licensing

Benefits

Compensation: annual salary of €60K-€100K + Employee stock option plan