Engineering Manager - Inference Performance
Hybrid (San Francisco), Hybrid (New York)$240k - $270kFull-timePosted 2w ago
About the role
Engineering Manager to lead part of the Inference Performance team focused on optimizing GPU-driven inference workloads. Hands-on technical leadership with people management, responsible for roadmap, execution, and team growth in a hybrid, multi-location setting.
Responsibilities
- Lead, mentor and grow a team of inference performance engineers through regular 1:1s, clear feedback, career development and performance reviews.
- Hire top GPU and inference engineering talent, and build a strong, collaborative team culture as the runtime team scales.
- Own the technical roadmap and execution for runtime performance work, balancing customer needs, new model launches and long-term platform investments.
- Stay close to the technical work. Review designs, guide profiling and optimization efforts, and help the team reason from first principles about where time and memory go.
- Drive the productionization of inference techniques such as quantization, speculative decoding, KV-cache reuse, chunked prefill and custom scheduling.
- Turn performance wins into measurable outcomes: tokens per GPU-hour, utilization, latency and cost.
- Help the team bring up and tune new model architectures on new hardware quickly, often in the same week they're released.
- Partner with Infrastructure, Inference Platform, Kernels, Model APIs and customer-facing teams to set priorities, coordinate launches and ship wins.
- Set high standards for engineering quality, benchmarking, operational excellence and incident response.
Qualifications
- Bachelor's, Master's, or Ph.D. degree in Computer Science, Engineering, Mathematics, or a related field.
- Experience managing engineers, including hiring, mentoring, giving feedback and running performance reviews.
- Experience leading or closely supporting GPU optimization teams in training, inference or recommendation systems.
- Strong technical depth in GPU workloads, with a solid understanding of GPU architecture and performance tradeoffs.
- Familiarity with ML libraries such as PyTorch, TensorRT or TensorRT-LLM.
- A track record of driving roadmaps and shipping complex technical projects with a team.
- Clear written and verbal communication, including the ability to align stakeholders across teams.
Skills
Required:People managementGPU optimizationGPU architecture & performanceMachine learning libraries
Benefits
- Competitive compensation, including meaningful equity
- (U.S. only) 100% coverage of medical, dental, and vision insurance for employee and dependents
- Flexible PTO policy including company wide Winter Break (office closed Christmas Eve to New Year’s Day)
- Paid parental leave
- Fertility and family-building stipend through Carrot
- (U.S. only) Company-facilitated 401(k)
- Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.
Last updated Oct 21, 2026. Interested? Apply on the company site.
About Baseten
Inference is everything
Effortlessly serve optimized open source & custom models on the fastest, most reliable model delivery network.
AICloud InfrastructureDeveloper ToolsInference PlatformMachine LearningModel DeploymentModel Runtimesai-infrastructure-and-development
More open roles at Baseten
- Partner Enablement Lead5d agoHybrid (San Francisco), Hybrid (New York) · G&A
- Engineering Manager Runtime - Storage5d agoHybrid (San Francisco) · EPD
- Sales Manager - Enterprise (New York)1w agoHybrid (New York) · GTM
- GRC Manager1w agoHybrid (San Francisco), Hybrid (Remote) +4 more · EPD
- Software Engineer - Inference Platform2w agoHybrid (San Francisco), Hybrid (Toronto) +3 more · EPD