Together AI is hiring a
Software Engineer, Inference Frameworks Engineer, Remote - United States

Summary

The job is for an ML Engineer focusing on optimizing large language model (LLM) inference frameworks at Together AI. The candidate should have relevant technical qualifications and experience in ML engineering optimization, GPU high-performance computing optimization, deep learning algorithms, and ML infrastructure. The company offers competitive compensation, health insurance, remote work flexibility, and other benefits.

Requirements

• Bachelor’s degree in Computer Science, Computer Engineering, or a relevant technical field, or equivalent practical experience
• Experience in ML engineering optimization is preferred
• Proficient in C/C++, Python, or Rust, with a strong understanding of algorithms and data structures
• Expertise in GPU high-performance computing optimization using CUDA, with an in-depth understanding of computer architecture
• Understanding of deep learning algorithms and neural network architectures
• Familiarity with TensorRT-LLM, ORCA, VLLM, and similar frameworks
• At least 3 years of experience working in ML infrastructure (e.g., PyTorch, SageMaker, etc.) and a solid understanding of deep learning training frameworks such as PyTorch and TensorFlow

Responsibilities

• Develop and optimize large language model (LLM) inference frameworks
• Optimize GPU and CUDA performance to create an industry-leading, high-performance LLM inference engine
• Design and implement distributed inference infrastructure for LLM
• Build monitoring and management tools to ensure the reliability and scalability of online inference servers
• Identify and resolve system inefficiencies and bottlenecks to improve overall system performance
• Develop tools to analyze bottlenecks and sources of instability, then design and implement solutions
• Collaborate with product teams to provide solutions that meet their requirements

Preferred Qualifications

• Knowledge of LLM models
• Familiarity with Rust programming

Benefits

• Competitive compensation
• Startup equity
• Health insurance
• Other benefits
• Flexibility in terms of remote work