← All roles

Principal Software Engineer, At-Scale Reliability and Fleet Intelligence — CSP Engagements

NVIDIA· Engineer, Sys SW
Apply on company site
Santa Clara, CA$272k - $431.3kfull-timePosted 3mo ago

About the role

Principal software engineer focused on fleet-scale reliability for CSP/hyperscale customers, driving MTBI-based reliability improvements across NVIDIA platforms by collaborating with CSP engineering teams, integrating fleet telemetry, and validating lab results in customer environments.

Responsibilities

  • Drive reliability work streams with CSP engineering teams — ensuring shared understanding of MTBI measurement methodology, failure classification, and health monitoring architecture
  • Gather and synthesize CSP fleet reliability data — identify failure patterns and champion improvements back into NVIDIA's firmware, driver, and hardware teams
  • Define consistent MTBI measurement methodology across CSP monitoring environments
  • Conduct fleet-scale failure pattern analysis using statistical methods (Pareto, survival analysis, Weibull) to classify failures as systemic, environmental, or configuration-specific
  • Drive fleet health monitoring integration architecture — ensure NVIDIA's health agents, telemetry, and reporting align with CSP workflows
  • Define burn-in reliability test environment and cluster certification criteria in collaboration with quality teams, validating criteria with customers
  • Collaborate with CSPs to ensure reliability-related integration work is complete ahead of at-scale launch
  • Develop predictive failure models using fleet telemetry and validate their effectiveness in customer environments

Qualifications

  • 15+ years of experience in systems software at datacenter scale, or reliability engineering with focus on at-scale challenges
  • BS or MS in Computer Science, Electrical Engineering, Statistics, or related field (or equivalent experience)
  • Deep expertise in multi-NUMA, rack-scale system software and firmware. Statistical failure analysis methods: MTBF/MTBI calculation, Pareto analysis, root cause classification
  • Experience with fleet-level telemetry and observability systems: time-series databases, anomaly detection, health scoring, event correlation
  • Understanding of hardware failure modes in large-scale GPU/accelerator deployments — ability to classify and prioritize across compute, interconnect, memory, power, and thermal domains
  • Experience defining or operating burn-in, stress testing, or certification frameworks for complex hardware systems. Familiarity with predictive maintenance or anomaly detection approaches applied to fleet health data
  • Customer obsession — genuine passion for understanding fleet reliability challenges at scale and translating them into actionable engineering priorities
  • Strong communication — ability to present statistical reliability findings to both deep technical audiences and executive leadership. Demonstrated success driving cross-functional improvements across hardware, firmware, and software teams without direct authority

Skills

Required:MTBI/MTBF analysisFleet telemetry and observabilityHardware failure mode analysisBurn-in/stress testing frameworksCommunication

Benefits

  • Equity
  • Benefits information available on NVIDIA careers site
Last updated Oct 23, 2026. Interested? Apply on the company site.

About NVIDIA

World Leader in AI Computing

NVIDIA invents the GPU and drives advances in AI, HPC, gaming, creative design, autonomous vehicles, and robotics.

AIautonomous vehiclesgaminggpugraphics cardhigh performance computingliquid-ai-recruiting-targetllm
Company profileWebsite

More open roles at NVIDIA