← All roles

Senior Software Engineer, DGX Cloud AI Infrastructure

NVIDIA· Engineer, Sys SW
Apply on company site
Hybrid (Santa Clara, CA), Hybrid (Austin, TX), Remote$184k - $287.5k + equityfull-timePosted 4mo ago

About the role

Senior Software Engineer to lead bring-up, triage, benchmarking, analysis, and optimization of distributed training and inference workloads across NVIDIA GPU platforms at scale. Sets technical direction across communication libraries, model frameworks, and inference/training stacks for state-of-the-art LLM workloads and builds resilience/failure-attribution capabilities for large clusters.

Responsibilities

  • Lead bring-up, validation, and debugging of large-scale AI clusters, infrastructure, and end-to-end workloads, setting the standard for how the team operates.
  • Bring up, tune, and benchmark AI pre-training, post-training, and inference workloads using PyTorch, NeMo / Megatron, TensorRT-LLM, and adjacent NVIDIA AI software stacks.
  • Profile and optimize end-to-end workload performance across compute, memory, networking, and communication layers using Nsight Systems, NCCL tests, and custom microbenchmarks.
  • Analyze scaling efficiency for distributed LLM workloads across modern GPU clusters, and translate findings into concrete tuning guidance.
  • Own root-cause analysis of complex failures — hangs, performance regressions, topology sensitivity in large distributed environments.
  • Define and build the resilience and failure-attribution stack: detecting, triaging, and attributing node, fabric, and workload failures across the cluster at scale.
  • Build repeatable benchmark suites, automation, acceptance criteria, and qualification workflows on new platforms.
  • Tune runtime settings, communication parameters, and deployment configurations in close partnership with framework, systems, and platform teams.
  • Deliver actionable, data-driven recommendations based on profiling, benchmark results, and cluster characterization.
  • Mentor engineers, drive technical standards, and act as a force multiplier across the broader performance and infrastructure organization.

Qualifications

  • Bachelor’s or Master’s in Computer Science or related field (or equivalent experience).
  • 8+ years of experience developing software infrastructure for large-scale AI or HPC systems with technical leadership.
  • Expertise debugging and triaging AI applications across full stack from application layer to hardware.
  • Deep hands-on experience with NCCL, CUDA-aware distributed execution, and debugging multi-GPU/multi-node workloads at scale.
  • Proven track record architecting, debugging, and scaling large-scale distributed systems.
  • Expert-level Python and C/C++ programming skills.
  • Experience operating workloads in scheduled, containerized cluster environments.
  • Excellent analytical, debugging, and communication skills with ability to influence across teams.

Skills

Required:PythonC/C++NCCLCUDAPyTorchDistributed systems

Benefits

  • Equity and benefits as described by NVIDIA benefits page.
Last updated Oct 2, 2026. Interested? Apply on the company site.

About NVIDIA

World Leader in AI Computing

NVIDIA invents the GPU and drives advances in AI, HPC, gaming, creative design, autonomous vehicles, and robotics.

AIautonomous vehiclesgaminggpugraphics cardhigh performance computingliquid-ai-recruiting-targetllm
Company profileWebsite

More open roles at NVIDIA