Software Engineer - Inference Performance
Hybrid (San Francisco), Hybrid (Toronto), Hybrid (New York), Hybrid (Montreal), Hybrid (Seattle)$180k - $360kFull-timePosted 2w ago
About the role
Inference performance engineer focused on making demanding AI workloads faster and more efficient across runtime, scheduling, serving, and routing with techniques like prefill/decode disaggregation, speculative decoding, and KV-cache management.
Responsibilities
- Implement and productionize cutting-edge inference techniques, working deep in runtime internals. That includes quantization, speculative decoding, KV-cache reuse, chunked prefill, LoRA, guided generation for structured outputs, and custom scheduling and routing algorithms.
- Profile and optimize inference end to end, from kernel launch overhead and memory layout up to request scheduling, prefill/decode disaggregation, and cache-aware routing. Run cross-layer investigations, such as tracing a tail-latency regression from request timing through routing and batching down to a kernel.
- Turn performance into cost savings. Improve tokens per GPU-hour, raise utilization, and give customers and internal teams clear latency/throughput/cost tradeoffs.
- Bring up and tune new model architectures on new hardware quickly, often in the same week they're released.
- Build benchmarking frameworks that measure real-world performance across model architectures, batch sizes, sequence lengths, and hardware configurations.
- Contribute upstream to open-source inference engines (vLLM, SGLang, TensorRT-LLM), and partner closely with model, infrastructure, and customer-facing teams to ship wins.
Qualifications
- Bachelor's, Master's, or Ph.D. degree in Computer Science, Engineering, Mathematics, or related field.
- Experience with one or more general-purpose programming languages, such as Python or C++.
- Familiarity with LLM optimization techniques (e.g., quantization, speculative decoding, continuous batching).
- Strong familiarity with ML libraries, especially PyTorch, TensorRT, or TensorRT-LLM.
- Demonstrated interest and experience in LLMs.
- Deep understanding of GPU architecture.
Skills
Required:QuantizationSpeculative decodingKV-cache optimizationGPU architecturePyTorch
Nice to have:PythonC++
Benefits
- Competitive compensation, including meaningful equity
- (U.S. only) 100% coverage of medical, dental, and vision insurance for employee and dependents
- Flexible PTO policy including company wide Winter Break (offices closed Christmas Eve to New Year’s Day)
- Paid parental leave
- Fertility and family-building stipend through Carrot
- (U.S. only) Company-facilitated 401(k)
- Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.
Last updated Oct 21, 2026. Interested? Apply on the company site.
About Baseten
Inference is everything
Effortlessly serve optimized open source & custom models on the fastest, most reliable model delivery network.
AICloud InfrastructureDeveloper ToolsInference PlatformMachine LearningModel DeploymentModel Runtimesai-infrastructure-and-development
More open roles at Baseten
- Partner Enablement Lead5d agoHybrid (San Francisco), Hybrid (New York) · G&A
- Engineering Manager Runtime - Storage5d agoHybrid (San Francisco) · EPD
- Sales Manager - Enterprise (New York)1w agoHybrid (New York) · GTM
- GRC Manager1w agoHybrid (San Francisco), Hybrid (Remote) +4 more · EPD
- Software Engineer - Inference Platform2w agoHybrid (San Francisco), Hybrid (Toronto) +3 more · EPD