← All roles

Senior Product Manager – AI Inference Performance

NVIDIA· Product Management
Apply on company site
Santa Clara, CA$208k - $327.8kPosted 2mo ago

About the role

Senior Product Manager responsible for the inference performance stack and related tooling across models and deployments on NVIDIA hardware, driving optimization techniques, benchmarking, and ecosystem integration to ensure best-in-class AI inference performance.

Responsibilities

  • Own the inference performance roadmap. Set direction across the stack: how models are represented, how memory and state are managed, how requests are scheduled and served, and how tokens get generated.
  • Build platforms, not one-offs. Deliver capabilities that generalize across model families, deployment topologies, and customer sizes. Build for easy adoption, sane defaults, and extensibility.
  • Agentic and Multi-Turn Workloads: Define the performance strategy for agentic applications, where long-running sessions, tool-call stalls, and unexpected output lengths break the assumptions built into single-turn serving. Drive capabilities around cross-turn cache reuse, request prioritization, and efficient handling of idle time in agent loops.
  • Framework & Ecosystem Strategy: Define how our optimizations land across TensorRT-LLM, vLLM, SGLang, and NVIDIA Dynamo. Partner with open-source communities and internal engineering teams so customers get great performance on NVIDIA hardware.
  • Benchmarking & Performance Claims: Own how performance is measured, published, and reproduced. Define the benchmark methodology, the metrics that matter (TTFT, ITL, throughput per GPU, cost per million tokens), and the guardrails that keep our numbers credible.
  • Run the product day to day. Own release readiness, quality bars, regression tracking, customer blocking issues, and the feedback loop from production deployments back into the roadmap.

Qualifications

  • 12+ years in product management at a technology company, or comparable time as a founder, engineering lead, or technical product owner.
  • Hands-on experience in AI/LLM Inference Performance Optimization in an engineering or product management capacity including deep familiarity with techniques like KV caching and reuse, quantization, speculative decoding, disaggregated serving
  • Familiarity with the inference and orchestration frameworks customers use: TensorRT-LLM, vLLM, SGLang, NVIDIA Dynamo, and the surrounding serving and scheduling ecosystem.
  • Proven track record of working independently — you can take an ambiguous problem space, define the strategy, and drive it to a shipped outcome without waiting to be told what to do next.
  • Operational experience running a live product: release management, quality and regression rigor, customer issues, and support processes.
  • Skill at translating low-level capability into business value — lower TCO, faster response, better GPU utilization — for engineers and executives alike.
  • BS, MS, or PhD in Computer Science, Computer Engineering, or another relevant area of study (or equivalent experience).

Skills

Required:Inference performance optimizationFrameworks and ecosystems familiarity

Benefits

  • Wide range of benefits information available at NVIDIA benefits site.
  • Eligibility for equity and benefits.
Last updated Oct 23, 2026. Interested? Apply on the company site.

About NVIDIA

World Leader in AI Computing

NVIDIA invents the GPU and drives advances in AI, HPC, gaming, creative design, autonomous vehicles, and robotics.

AIautonomous vehiclesgaminggpugraphics cardhigh performance computingliquid-ai-recruiting-targetllm
Company profileWebsite

More open roles at NVIDIA