Staff Software Engineer, Online Database Infrastructure - Ingestion

Summary

Join Airbnb's Ingestion team and become a resident expert on the online data exports platform. Collaborate with internal product teams to effectively utilize the platform, building and operating a data ingestion system enabling offline access and processing of online data. You will be hands-on in code, design, and testing, working with cross-team partners on multi-month projects. Lead a team of developers on multi-quarter projects, stay updated on data ingestion systems, and incorporate new technologies. Mentor team members and raise operational standards by proactively identifying and fixing issues. Be part of the on-call rotation.

Requirements

• 9+ years of experience building and operating large scale core backend distributed systems
• Hands on experience building and operating large scale online distributed databases and offline data ingestion pipelines to export data from underlying storage. Familiarity with internals of distributed databases and offline data warehouse systems is a must
• Strong cross team collaboration and written and verbal communication skills and capable of influencing stakeholders and conveying complex technical concepts effectively
• Ability to provide technical leadership, mentorship and guiding teams on best practices around data ingestion systems
• Ability to own and dive deeply in a complex code base and knack for writing clean, readable, testable, maintainable code
• Considerable experience working in either Java, Scala or Python and familiarity with SPARK and Kafka
• Demonstrate strong ownership and ability to effectively plan multi month projects and deliver them in a timely manner

Responsibilities

• Be an expert in building and operating a data ingestion system that enables offline accessing and processing of online data at Airbnb in the form of (but not limited to) DB snapshots into the warehouse and near real time change data capture (CDC) events
• Be hands-on (code, design, test) and collaborate with cross team partners (internal customers, dependencies and leadership) to deliver on multi-month projects in a timely fashion
• Lead a team of developers to deliver multi-quarter cross functional projects
• Stay up-to-date with the latest developments in data ingestion systems and evaluate and incorporate new technologies and methodologies to improve the existing data ingestion architecture/solution
• Influence the team’s and organization’s long-term roadmap and strategy
• Mentor and coach team members, providing guidance and support to enhance their skills and technical standards in the team
• Raise operational standards by effectively and proactively identifying, debugging and fixing operational issues. Be part of the oncall rotation for the Ingestion team

Preferred Qualifications

• Experience with AWS and/or GCP
• Experience working with Flink, K8s, Airflow, AWS Aurora or TiDB
• Experience working with Iceberg and generating incremental Iceberg DB snapshots

Benefits

• Bonus
• Equity
• Benefits
• Employee Travel Credits