Remote Software Engineer

Summary

The job is for a Data Platform Engineer at Stack AV who will design, implement, and maintain the on-premises data platform. The candidate should have experience with large scale interactive and batch workloads, Kubernetes compute, S3 compatible object stores, Data Engineering concepts, open source technologies, streaming platforms, processing engines, table formats, Python, SQL, and workflow orchestration systems.

Requirements

• Experience supporting large scale interactive and batch workloads
• Experience with on-premises infrastructure, Kubernetes compute, S3 compatible object stores (Vast Data, Minio) managed via Terraform or ArgoCD
• Expert level understanding of Data Engineering concepts and open source technologies
• Experience running a modern data stack on-premises using open source technologies
• Familiarity with streaming platforms (Kafka, Pulsar), processing engines (Trino, Apache Spark), table formats (ICEBERG, Delta Lake, Hudi, Xtable) and Postgres
• Fluent in Python, and SQL, with experience writing and maintaining highly available data applications using Trino and Apache Spark
• Strong focus on requirements gathering and overall design process
• Experience architecting and optimizing large-scale systems to facilitate machine learning, simulation, and data processing exercises
• Experience running and supporting workflow orchestration systems such as Argo Workflows, Airflow, or Flyte.io
• Strong focus on providing production-grade service to internal engineering teams through observability, instrumentation, and well-defined SLOs
• Experience working in a diverse and distributed team, with a focus on communication and customer empathy

Responsibilities

Design, implement, and maintain the Stack AV on-premises data platform