Software Engineer, Cloud Inference

Summary

Join Modular's Cloud Inference team and help revolutionize AI infrastructure. We're building end-to-end distributed LLM inference deployments integrated with the MAX stack. This role requires experience in backend engineering, AI inference, and distributed systems. You will collaborate with product and engineering teams, design and develop helm charts and cloud services, and contribute to achieving SOTA performance. We offer competitive compensation, including stock options, and world-class benefits such as premier insurance plans, 401k matching, and flexible paid time off. Remote work options are available for US and Canada-based candidates.

Requirements

• Experience working on high scale ML inference infrastructure (traditional AI or genAI)
• Experience with kubernetes
• Familiarity with HuggingFace API and workflows for using community models
• Ability to create durable, reusable software tools and libraries that are leveraged across teams and functions
• Experience in machine learning technologies and use cases
• Creativity and curiosity for solving complex problems, a team-oriented attitude that enables you to work well with others, and alignment with our culture
• Strongly identifies with our core company cultural values

Responsibilities

• Work with Product and partner engineering teams to design and ship new inference server features
• Collaborate with our kernels and genAI teams to achieve SOTA performance at the serving layer
• Help design and develop helm charts and cloud services for scaled LLM inference (intelligent routing, distributed kvcache management, disaggregated inference, etc.)

Preferred Qualifications

• Experience building ML models in PyTorch
• Familiarity with modern C++
• Some experience with and interest in Mojo, our AI programming language!

Benefits

• Premier insurance plans
• Up to 5% 401k matching
• Flexible paid time off
• Competitive Compensation
• Stock options
• Team Building Events
• Remote work options (US and Canada)