Staff Software Engineer

Summary

Join the team as Twilio's next Staff Software Engineer - Profiles. Design and develop robust, scalable data pipelines using Databricks, Apache Spark, and Delta Lake. Collaborate with cross-functional teams to gather requirements and design data solutions that meet business needs.

Requirements

• 7+ years of experience in software engineering, with a focus on data engineering and lake house architecture
• Strong proficiency in Spark and Scala; experience with Databricks technologies is highly desirable
• Strong proficiency with cloud platforms (e.g., AWS, GCP) and data storage solutions (e.g., Delta Lake, Iceberg)
• Solid understanding of ETL processes and data pipeline architecture
• Experience with SQL and data modeling; knowledge of NoSQL databases is a plus
• Fundamental understanding and experience with asynchronous architectures, their benefits and challenges including various broker implementations (Kafka, SQS, MQ)
• Experience operating large-scale, distributed systems on top of cloud infrastructure such as Amazon Web Services (AWS) or Google Compute Platform (GCP)
• Have hands-on experience with container orchestration frameworks (e.g. Kubernetes, EKS, ECS)
• Experience shipping services (products) following CI/CD development paradigm
• Deep understanding of architectural patterns of high-scale web application (e.g., well-designed APIs, high volume data pipelines, efficient algorithms)
• Have led teams, large projects, or owned and built an important, complex system end to end
• Excellent written and verbal communication skills

Responsibilities

• Design and develop robust, scalable data pipelines using Databricks, Apache Spark, and Delta Lake
• Collaborate with cross-functional teams to gather requirements and design data solutions that meet business needs
• Optimize performance and scalability of data pipelines, ensuring high availability and reliability
• Ensure data quality, consistency, and security across the data platforms
• Monitor and troubleshoot data workflows and systems, providing timely resolutions to issues
• Mentor and guide junior engineers, promoting best practices in software development and data engineering
• Stay current with industry trends and technologies to continuously improve our data architecture
• Build globally available and highly scalable distributed systems
• Participate in an on-call rotation to support our business-critical infrastructure

Benefits

• Generous time-off
• Ample parental and wellness leave
• Healthcare
• A retirement savings program