← All roles

(Staff / Senior Staff) AI Infra Engineer, Inference Infrastructure

Together AI· Engineering
Apply on company site
San Francisco, CA$250k - $320k + equity + bonusfull-timePosted 5d ago

About the role

Production-focused AI infrastructure engineer responsible for the Kubernetes-native control plane that provisions and runs a GPU inference fleet, owning reliability, SLOs, and incident response for inference serving and capacity efficiency.

Responsibilities

  • Operate the provisioning state machine in production: health of the host lifecycle pipeline, discovery, inference bring-up, GPU driver/CUDA validation, health checks, decommission/RMA, triaging stuck or failed state transitions at fleet scale and feeding real failure patterns back into how the states and transitions are designed.
  • Triage and extend automated self-healing: diagnose new failure modes live, drive fixes to production, and turn into new detection/remediation plugins for automatic future handling.
  • Own end-to-end reliability of the pipeline: hold SLOs and error budgets for provisioning and fleet health, lead incident response, drive systemic fixes rather than hotfixes.
  • Partner with the inference/ML platform team from the production side: understand real cluster topologies, interconnects, and scheduling constraints and translate failures into platform requirements.
  • Work with ML researchers, engineers, and product managers to bring latest technologies into production, enabling advanced research through engineering.
  • Introduce new techniques, tools, and architecture to improve performance, latency, throughput, and efficiency of the model inference stack.
  • Build tools for visibility into bottlenecks and instability, and implement solutions for high-priority issues.
  • Lead reliability cross-functionally: coordinate infra, ML platform, and product teams around reliability priorities, manage stakeholders through incidents and roadmap trade-offs.

Qualifications

  • Experience operating a provisioning or host-lifecycle pipeline in production, troubleshooting stuck or failed state transitions at fleet scale, and integrating real failure patterns into state machine design.
  • Deep experience owning production reliability for large-scale distributed systems; GPU/ML-serving infrastructure experience is a strong plus.
  • Hands-on experience troubleshooting InfiniBand/RDMA networking issues, comfortable reading IB port counters and subnet manager logs.
  • Familiarity with multi-tenant model serving (LoRA/adapter-based serving) and running differentiated SLOs across model pools of different priority.
  • Track record of eliminating operational toil through automation and tooling you built yourself.
  • Hands-on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node-level health/remediation systems.
  • Comfort with event-driven architectures, backpressure/circuit-breaking design, and built-in observability (metrics, logs, traces).
  • Software-engineering mindset: write and ship production code; treat GPU infrastructure as software to be engineered.

Skills

Required:Provisioning/host-lifecycle pipeline managementInfiniBand/RDMA troubleshootingKubernetes controller/CRD patterns
Nice to have:GPU/ML-serving infrastructure

Benefits

  • health insurance
  • startup equity
  • competitive benefits
Last updated Oct 21, 2026. Interested? Apply on the company site.

About Together AI

The AI Acceleration Cloud - Fast Inference, Fine-Tuning & Training

Run and fine-tune generative AI models with simple APIs and scalable GPU clusters. Train & deploy at scale on The AI Acceleration Cloud.

AI native cloudFine-TuningGPU clustersInferenceTrainingai-infrastructure-and-developmentbrex-fastest-growing-2025liquid-ai-recruiting-target
Company profileWebsite

More open roles at Together AI