Performance Engineer, Inference
Bengaluru, ChennaiFull-timePosted 2mo ago
About the role
Senior Performance Engineer focusing on production serving path for large distributed models, integrating model/kernel artifacts into a multi-node, multi-tenant serving stack, and delivering latency/throughput and cost metrics.
Responsibilities
- Own Sarvam's production serving path for large distributed models end to end.
- Read/modify and operate distributed-serving stacks (SGLang, vLLM, NVIDIA Dynamo, TensorRT-LLM) as needed for workload fit.
- Integrate artifacts from model and kernel teams into a running multi-node, multi-tenant stack.
- Build and train speculators (draft models, distillation, acceptance-rate tuning) for live serving distributions.
- Produce and defend latency and throughput numbers the company plans against; collaborate across teams (architecture, kernels, model, SRE).
- Maintain on-call ownership of an inference SLO.
Qualifications
- 5+ years in ML systems, with 2+ years on inference serving at production scale.
- Experience serving 100B+ parameter models in production across multi-node tensor, pipeline, or expert parallelism.
- Source-level fluency in SGLang, vLLM, Dynamo, or TensorRT-LLM and familiarity with the other three.
- Experience with disaggregated prefill-decode, distributed KV/cache transfer, and cross-node routing/scheduling.
- Experience with speculative decoding and training/evaluating draft models or speculators.
- Deep understanding of KV cache internals (block tables, copy-on-write, prefix sharing, fragmentation).
- Working knowledge of TP/PP/EP, NCCL primitives and scheduler interactions.
- Multi-tenant serving, model co-location and MIG/MPS isolation.
- Profiling fluency (Nsight Systems, framework tracing, py-spy/perf).
- C++ and CUDA at read-and-modify level.
- On-call ownership of an inference SLO.
Skills
Required:ML systems performance engineeringSGLang / vLLM / Dynamo / TensorRT-LLM
Last updated Aug 11, 2026. Interested? Apply on the company site.
About Sarvam
Sarvam | India's Full-Stack Sovereign AI Platform
Sarvam is India's full-stack sovereign AI platform, with speech-to-text, text-to-speech, translation, and conversational agents across 22 Indian languages. Start free.
API PricingAbout UsAksharAryaCareersConversationsDocument DigitisationDocumentation
More open roles at Sarvam
- Software Engineer - Model API Platform1w agoBengaluru · Engineering
- Staff Engineer - Systems Engineering2w agoBengaluru · Engineering
- Staff Engineer - Telephony2w agoBengaluru · Engineering
- Growth Marketing Intern3w agoBengaluru · Marketing
- Marketing Intern4w agoBengaluru · Marketing