Senior Backend Engineer, Inference Platform
Together AI· Engineering
Hybrid (San Francisco)$200k - $290kfull-timePosted 11mo ago
About the role
Senior backend role focused on building and optimizing the core inference backbone for Together AI’s multi-tenant, serverless inference platform, collaborating with researchers and open source communities to push frontier models into production at scale.
Responsibilities
- Build and optimize global and local request routing, ensuring low-latency load balancing across data centers and model engine pods
- Develop auto-scaling systems to dynamically allocate resources and meet strict SLOs across dozens of data centers
- Design systems for multi-tenant traffic shaping, tuning both resource allocation and request handling — including smart rate limiting and regulation — to ensure fairness and consistent experience across all users
- Engineer trade-offs between latency and throughput to serve diverse workloads efficiently
- Optimize prefix caching to reduce model compute and speed up responses
- Collaborate with ML researchers to bring new model architectures into production at scale
- Continuously profile and analyze system-level performance to identify bottlenecks and implement optimizations
Qualifications
- 5+ years of demonstrated experience building large-scale, fault-tolerant, distributed systems and API microservices
- Strong background in designing, analyzing, and improving efficiency, scalability, and stability of complex systems
- Excellent understanding of low-level OS concepts: multi-threading, memory management, networking, and storage performance
- Expert-level programming in one or more of: Rust, Go, Python, or TypeScript
- Knowledge of modern LLMs and generative models and how they are served in production is a plus
- Experience with open source ecosystem around inference; familiarity with SGLang, vLLM, or NVIDIA Dynamo is especially handy
- Experience with Kubernetes or container orchestration is a strong plus
- Familiarity with GPU software stacks (CUDA, Triton, NCCL) and HPC technologies (InfiniBand, NVLink, MPI) is a plus
- Bachelor’s or Master’s degree in Computer Science, Computer Engineering, or related field, or equivalent practical experience
Skills
Required:Distributed systemsPerformance optimizationProgramming languages
Nice to have:Container orchestrationGPU and HPC technologies
Benefits
- Competitive compensation, equity, and benefits
- Health insurance and other competitive benefits
Last updated Sep 9, 2026. Interested? Apply on the company site.
About Together AI
The AI Acceleration Cloud - Fast Inference, Fine-Tuning & Training
Run and fine-tune generative AI models with simple APIs and scalable GPU clusters. Train & deploy at scale on The AI Acceleration Cloud.
AI native cloudFine-TuningGPU clustersInferenceTrainingai-infrastructure-and-developmentbrex-fastest-growing-2025liquid-ai-recruiting-target
More open roles at Together AI
- Research Intern, Frontier Speculative Decoding (Summer 2027)4d agoHybrid (San Francisco, CA) · Research
- Research Intern, Frontier Speculative Decoding (Winter 2027)4d agoHybrid (San Francisco) · Research
- Director of Investor Relations4d agoHybrid (San Francisco, CA) · Finance
- Recruiting Sourcer, Business (GTM, Infrastructure Strategy & G&A)4d agoHybrid (San Francisco) · People
- Staff Software Engineer - AI Compute, Together Cloud4d agoAmsterdam · Engineering