← All roles

Training Performance Engineer

OpenAI· Scaling
Apply on company site
Hybrid (San Francisco)$295k - $500kFull-timePosted 11mo ago

About the role

Role focusing on driving efficiency improvements across OpenAI's distributed training stack, analyzing large-scale runs, identifying utilization gaps, and designing optimizations to push throughput and uptime.

Responsibilities

  • Profile end-to-end training runs to identify performance bottlenecks across compute, communication, and storage.
  • Optimize GPU utilization and throughput for large-scale distributed model training.
  • Collaborate with runtime and systems engineers to improve kernel efficiency, scheduling, and collective communication performance.
  • Implement model graph transforms to improve end to end throughput.
  • Build tooling to monitor and visualize MFU, throughput, and uptime across clusters.
  • Partner with researchers to ensure new model architectures scale efficiently during pre-training.
  • Contribute to infrastructure decisions that improve reliability and efficiency of large training jobs.

Qualifications

  • Love optimizing performance and digging into systems to understand how every layer interacts.
  • Strong programming skills in Python and C++ (Rust or CUDA a plus).
  • Experience running distributed training jobs on multi-GPU systems or HPC clusters.
  • Enjoy debugging complex distributed systems and measuring efficiency rigorously.
  • Exposure to frameworks like PyTorch, JAX, or TensorFlow and an understanding of how large-scale training loops are built.
  • Comfortable collaborating across teams and translating raw profiling data into practical engineering improvements.

Skills

Required:Performance profiling and optimizationGPU utilization and throughput optimizationDistributed trainingPythonC++PyTorch/JAX/TensorFlow
Nice to have:CUDA

Benefits

  • Medical, dental, and vision insurance for you and your family, with employer contributions to Health Savings Accounts
  • Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses (parking and transit)
  • 401(k) retirement plan with employer match
  • Paid parental leave (up to 24 weeks for birth parents and 20 weeks for non-birthing parents), plus paid medical and caregiver leave (up to 8 weeks)
  • Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees
  • 13+ paid company holidays, and multiple paid coordinated company office closures throughout the year for focus and recharge, plus paid sick or safe time
  • Mental health and wellness support
  • Employer-paid basic life and disability coverage
  • Annual learning and development stipend to fuel your professional growth
  • Daily meals in our offices, and meal delivery credits as eligible
  • Relocation support for eligible employees
  • Additional taxable fringe benefits, such as charitable donation matching and wellness stipends, may also be provided.
Last updated Sep 3, 2026. Interested? Apply on the company site.

About OpenAI

ai-infrastructure-and-developmentliquid-ai-recruiting-targetllmsetter-30setter-30-2026-q1speech-to-texttext-to-speechvoice-ai-stack
Company profileWebsite

More open roles at OpenAI