← All roles

[Junior / Senior / Staff] Software Engineer, Inference / Compute Infrastructure Engineering

Together AI· Engineering
Apply on company site
Hybrid (San Francisco)$240k - $280k + equityfull-timePosted 3mo ago

About the role

Software engineer to build Kubernetes-native control plane for provisioning and running GPU inference fleet, with declarative APIs, controllers, and scheduling/workload optimization to maximize GPU utilization and simplify platform use for inference teams.

Responsibilities

  • Build the provisioning state machine: model full lifecycle of a physical host from discovery, inference bring-up to GPU driver/CUDA stack, health validation, and decommission/RMA — as explicit, versioned states and transitions.
  • Build the self-service API: design declarative APIs and a control plane so the inference team can request, scale, and tear down inference clusters with one API call — no ticket, no human in the loop.
  • Automate self-healing: detect degraded or failed nodes, drain them safely, trigger repair or replacement, and reintroduce healthy capacity into the pool automatically.
  • Own reliability of the pipeline: idempotency, retries, rollback, and drift detection so the provisioning system is as dependable as any other production service.
  • Partner with the inference/ML platform team: understand the cluster shapes they need — topology, interconnect, scheduling constraints — and encode them as first-class abstractions in the platform.
  • Engineer it like software: strong typing, automated tests, code review, versioning, and CI/CD for infrastructure code — this is a product, not a collection of Ansible playbooks.

Qualifications

  • Strong software engineering background in Go, Python, Rust, or similar — you write and test real software for a living.
  • Experience with durable workflow orchestration tools such as Temporal, Cadence, or equivalent to run long-lived, manifest-driven workflows that survive failures and resume mid-execution.
  • Experience building software control planes or orchestration systems that model state and reconcile it over time (e.g., Kubernetes controllers/operators, custom reconciliation loops, workflow engines).
  • Experience with event-driven systems — designing and building software around message queues, event streams, or pub/sub (e.g., Kafka, NATS, SQS) rather than polling or cron-driven scripts.
  • A product mindset. You’ve built internal platforms or APIs consumed by other engineering teams and care about the developer experience of what you ship.

Skills

Required:GoPythonKubernetesTemporal / CadenceEvent-driven architectures
Nice to have:Rust

Benefits

  • competitive compensation
  • startup equity
  • health insurance and other competitive benefits
Last updated Sep 23, 2026. Interested? Apply on the company site.

About Together AI

The AI Acceleration Cloud - Fast Inference, Fine-Tuning & Training

Run and fine-tune generative AI models with simple APIs and scalable GPU clusters. Train & deploy at scale on The AI Acceleration Cloud.

AI native cloudFine-TuningGPU clustersInferenceTrainingai-infrastructure-and-developmentbrex-fastest-growing-2025liquid-ai-recruiting-target
Company profileWebsite

More open roles at Together AI