← All roles

Staff Software Engineer - AI Compute, Together Cloud

Together AI· Engineering
Apply on company site
San Francisco, CA$260k - $300kfull-timePosted 3w ago

About the role

Staff Software Engineer focusing on AI Compute in the Together Cloud org, responsible for architecting and building the next-generation AI cloud platform, including GPU virtualization, in-DC IaaS, and global management planes for on-demand and reserved Kubernetes/Slurm clusters across data centers and thousands of GPUs.

Responsibilities

  • Own the GPU and network virtualization stack: the hypervisor, kernel, and SDN work that keeps GPU compute and DC networking high-performance, portable, and strongly isolated across heterogeneous hardware.
  • Own the in-DC IaaS layer: architect and roadmap the services, Kubernetes operators, and libraries that provision and manage compute, storage, and networks in our data centers — VMs, parallel filesystems, VPCs, and InfiniBand partitions; lead its build-out for a new Vera Rubin data center with thousands of GPUs, from hardware bring-up to customer-facing API.
  • Design the GPU scheduling and global management plane: the distributed control plane behind on-demand and reserved clusters across dozens of data centers, including the systems that scale per-cluster limits and automate the onboarding of new capacity.
  • Architect monitoring and automated remediation for fault tolerance: the strategy for automated detection, isolation, and recovery of failed nodes that keeps distributed pretraining and large-scale inference running through hardware failures.
  • Set technical direction across teams: lead design reviews, resolve cross-cutting architectural disagreements, unblock cross-team dependencies and integration risks, and define the standards other engineers build against — measured in cluster reliability, time-to-first-GPU on new capacity, and quality at scale.
  • Grow the team: mentor senior and junior engineers, deepen the team's expertise in virtualization, DC networking, and GPU infrastructure, and help raise the hiring bar for Together Cloud.
  • Set the engineering bar: create the testing frameworks, tools, and developer documentation that make our systems robust and usable by other teams, and shape the core, open-source Together AI platform.

Qualifications

  • 7+ years of professional software development experience, with expert-level proficiency in at least one backend language (Golang desired), writing high-performance, well-tested, production-quality code.
  • Track record of owning the architecture of large distributed systems from blank page to production at scale, including the judgment calls that could not be reversed cheaply.
  • Deep experience building and operating globally distributed, high-performance microservice architectures across one or more cloud providers (AWS, Azure, GCP).
  • Expert systems knowledge across compute, networking, and storage — including concurrency, memory management, performant I/O, and scale at a global level.
  • Demonstrated technical leadership beyond your own commits: mentoring senior engineers, leading design reviews, and driving alignment across teams that do not report to you.
  • Excellent communication and diplomacy skills — able to write design docs that settle arguments, and to work effectively with technical and non-technical stakeholders.
  • Experience building and operating reliable, customer-facing production systems at scale, and owning the infrastructure automation (Terraform, Ansible), observability (Prometheus, Grafana), and CI/CD (GitHub Actions, ArgoCD) that keep them healthy.

Skills

Required:Backend developmentDistributed systems architectureCloud and microservices

Benefits

  • health insurance
  • startup equity
  • remote-flexible work options
  • competitve compensation with equity and benefits
Last updated Sep 30, 2026. Interested? Apply on the company site.

About Together AI

The AI Acceleration Cloud - Fast Inference, Fine-Tuning & Training

Run and fine-tune generative AI models with simple APIs and scalable GPU clusters. Train & deploy at scale on The AI Acceleration Cloud.

AI native cloudFine-TuningGPU clustersInferenceTrainingai-infrastructure-and-developmentbrex-fastest-growing-2025liquid-ai-recruiting-target
Company profileWebsite

More open roles at Together AI