Senior HPC Engineer - Fleet Engineering
Lambda· Data Center Business
Hybrid (San Francisco Office (Fremont St)), Hybrid (San Jose Office (First St)), Hybrid (Remote, USA), Hybrid (Bellevue Office)$267k - $356kFull-timePosted 2mo ago
About the role
Senior HPC Engineer responsible for building and operating large-scale HPC fleets for AI workloads, with emphasis on monitoring, automation, incident response, and collaboration with on-site deployment teams.
Responsibilities
- Build and operate monitoring and alerting for cluster health — fabric, GPU, power/thermal, and job-level signals — to detect and respond to issues proactively
- Remotely deploy and configure large-scale HPC clusters for AI workloads using automation wherever possible
- Automate cluster lifecycle: operating systems, firmware, drivers, and networking, managed as code (Ansible, Terraform) rather than by hand
- Create runbooks and automated remediations for common cluster failure modes, designed so Support and HPC Support can run them safely
- Troubleshoot and resolve cluster issues across InfiniBand/RoCE, NCCL, GPU-direct, fabric, switching, and power — working closely with on-site deployment teams
- Participate in on-call rotations and lead incident response for cluster-level problems
- Contribute to and maintain Standard Operating Procedures, and feed clear requirements back to other engineering teams on simplification, stability, and operational efficiency
Qualifications
- 7+ years of experience in Site Reliability Engineering, HPC Engineering, DevOps, or a similar role
- Have a strong understanding of modern AI infrastructure, from GPU architectures to hardware performance optimization
- Strong understanding of Linux-based systems in a distributed environment
- Are experienced configuring and troubleshooting InfiniBand (IB), RoCE, CLOS fabrics, 100GbE, Ethernet/switching, GPU-direct, and NCCL environments
- Solid understanding of Python and Go, with experience working with SWE teams to improve internal tooling
- Experience with monitoring and alerting tools (e.g., Prometheus, Grafana, Clickhouse)
- Proficiency in automation and configuration management tools (e.g., Ansible, Terraform)
- Have excellent problem-solving and troubleshooting skills and an innate attention to detail
- Passion for continuous improvement and innovation
Skills
Required:Monitoring and alertingAutomation and configuration managementDistributed Linux systemsNetworking and HPC fabricsProgramming
Benefits
- Health, dental, and vision coverage for you and your dependents
- Wellness and commuter stipends for select roles
- 401k Plan with 2% company match (USA employees)
- Flexible paid time off plan that we all actually use
Last updated Sep 24, 2026. Interested? Apply on the company site.
About Lambda
The Superintelligence Cloud
The gigawatt-scale AI GPU Cloud built for superintelligence. Featuring on-demand & reserved cloud NVIDIA HGX B200, B300, GB200, GB300 and H200 GPUs for AI training & inference.
AI trainingDeep learningGPU CloudInferenceNVIDIA GPUsliquid-ai-recruiting-targetsetter-30setter-30-2026-q1
More open roles at Lambda
- Senior Workday Integration Architect4d agoHybrid (San Francisco Office (Fremont St)), Hybrid (San Jose Office (First St)) · Data Center Business
- Workplace Coordinator4d agoSan Jose Office (First St), San Jose Office (Zanker) · G&A
- Employment Counsel4d agoHybrid (San Francisco Office (Second St)), Hybrid (New York, NY) · G&A
- Head of Tax4d agoHybrid (San Francisco Office (Second St)), Hybrid (San Jose Office (First St)) +1 more · G&A
- Controller4d agoHybrid (San Jose Office (First St)), Hybrid (San Francisco Office (Second St)) · G&A