← All roles

Helix AI Engineer, Training Performance

Figure· AI - Helix Team
Apply on company site
San Jose, CA$200k - $400kfull-timePosted 2mo ago

About the role

AI Training Performance Engineer focused on improving distributed training frameworks for large-scale model training, optimizing GPU kernels, and co-designing models to maximize hardware utilization for 100B+ parameter models across 100k+ GPUs.

Responsibilities

  • Optimize training performance for a 100B+ parameter models across 100k+ GPUs.
  • Collaborate on accelerator choice, cluster topology, scheduling, and hardware procurement to inform scaling.
  • Write and optimize custom kernels (Triton/CUDA).
  • Build tooling and dashboards for continuous performance monitoring, regression detection, and root-cause analysis across training jobs.
  • Optimize data loading and preprocessing pipelines so I/O never gates accelerators.
  • Improve checkpointing, fault tolerance, and elastic restart for quick recovery from node failures without significant wall-clock time loss.
  • Partner with researchers to co-design model architectures/training recipes performant at scale (activation checkpointing, mixed precision, sequence packing).
  • Extend/contribute to kernel compilers (e.g., Triton, Gluon) to improve iteration speed and target custom/non-NVIDIA accelerators.
  • Build/extend agentic systems that generate, benchmark, and iterate on custom kernels.
  • Evaluate emerging accelerator architectures (AMD, TPU, SRAM-based ASICs, etc.) for fit with training workloads; lead proof-of-concept ports/benchmarks.
  • Explore different model/data parallelisms (FSDP, context parallel, expert parallel) to determine optimal configuration per model size.

Qualifications

  • Bachelor's or Master's degree in Computer Science, Computer/Electrical Engineering, or related field.
  • 3+ years in AI performance engineering with leadership on large-scale performance improvement projects.
  • Deep understanding of GPU architecture and performance characteristics (memory bandwidth, compute-bound vs. memory-bound ops, occupancy).
  • Proficiency with profiling tools (Nsight Systems/Compute, PyTorch Profiler, HTA, or similar) and ability to translate traces into concrete optimizations.
  • Solid grasp of collective communication (NCCL) and modern networking concepts (RDMA, NVLink, InfiniBand/RoCE, topology-aware placement).
  • Strong Python and CUDA/C++ skills; comfortable reading/modifying framework internals.
  • Experience debugging performance regressions and instability at scale (stragglers, hangs, OOMs, numerical divergence).
  • Experience defining/reasoning about hardware-efficiency metrics (MFU/HFU) and using them to drive optimization priorities.

Skills

Required:GPU performance optimization

Benefits

  • Total compensation package may include benefits depending on role (details shared with offer).
Last updated Oct 7, 2026. Interested? Apply on the company site.

About Figure

The future of home help is here

Figure is the first-of-its-kind AI robotics company bringing a general purpose humanoid to life.

liquid-ai-recruiting-targetphysical-ai-funding-landscapesetter-30setter-30-2026-q1
Company profileWebsite

More open roles at Figure