Infrastructure, Large-scale Training
San Jose, CA$180k - $450kfull-timePosted 7mo ago
About the role
Leading and managing large-scale GPU computing clusters powering AI training and deployment workloads. Combine systems engineering with ML infrastructure to ensure reliability, scalability, and efficiency of the compute platform for research and engineering teams.
Responsibilities
- Design, implement, and maintain Infrastructure as Code (IaC) best practices to enable repeatable, auditable, and scalable cluster provisioning.
- Enhance and harden CI/CD deployment pipelines to ensure robust, secure, and low-latency model service delivery across production environments.
- Own and evolve stable training infrastructure operating at the scale of 10,000+ GPUs, including job scheduling, fault tolerance, and network fabric optimization.
- Partner closely with ML researchers and engineers to understand compute bottlenecks and translate them into infrastructure improvements.
- Monitor system health, define SLOs, and lead incident response for critical training and inference workloads.
- Drive capacity planning, cost efficiency initiatives, and hardware lifecycle management across the GPU fleet.
- Contribute to internal tooling and platform abstractions that improve developer experience for teams consuming compute resources.
Qualifications
- 5+ years of experience in infrastructure, systems, or platform engineering, with at least 2 years working in ML or HPC environments.
- Demonstrated experience managing GPU clusters or large-scale distributed compute infrastructure.
- Strong proficiency in at least one systems or infrastructure programming language.
- Deep understanding of networking fundamentals (RDMA, InfiniBand, or RoCE a plus) relevant to high-throughput training workloads.
- Experience with container orchestration, job scheduling, and multi-tenant resource management.
- Proven track record owning production systems with high reliability requirements.
- Strong debugging and observability skills across the full infrastructure stack.
Skills
Required:Infrastructure as Code (IaC) practicesCI/CD pipelinesGPU cluster managementJob scheduling and fault toleranceNetworking fundamentalsContainer orchestration
Last updated Jul 16, 2026. Interested? Apply on the company site.
About Hark
building the most advanced personal intelligence in the world.
At Hark, we are building the most advanced personal intelligence in the world.
personal intelligenceartificial intelligence
More open roles at Hark
- Warehouse Associate5d agoSan Jose, CA · Supply Chain
- Industrial Designer1w agoHybrid (San Jose, CA) · Design
- Member of Technical Staff, LLM & Production1w agoSan Jose · AI Foundation Models
- Member of Technical Staff, Hark Desktop2w agoHybrid (San Jose) · Product Engineering
- Application Security Engineer2w agoSan Jose, CA · Privacy & Security