Software Engineer - GPU Networking & Distributed Systems
Hybrid (San Francisco), Hybrid (Toronto), Hybrid (New York), Hybrid (Montreal), Hybrid (Seattle)$165k - $330kFull-timePosted 8mo ago
About the role
Foundation engineer to lead GPU networking efforts, architecting software for disaggregated serving and distributed inference across large GPU clusters with RDMA-first design and high-performance interconnects.
Responsibilities
- Make RDMA First-Class: integrate RDMA/RoCE/InfiniBand into the inference stack to unlock higher bandwidth and lower latency.
- Optimize Distributed Inference: implement and tune networking layers for Disaggregated KV Cache Offload and WideEP across NVLink and InfiniBand for MoE models.
- Enable Serverless-Grade Startup Speeds for LLMs: work on checkpointing and storage to enable sub-10-second startup for trillion-parameter models.
- Deep-Dive into Hardware: characterize and validate networking performance on bleeding-edge clusters (H100/H200, B200/B300, NVL72), write acceptance tests for peak throughput and low latency.
- Build Observability: design tools to visualize packet flow, congestion, and effective bandwidth across GPU interconnects.
- Optimize Kernels: work with NCCL, NVSHMEM and potentially write custom communication kernels to overlap compute and data transfer.
Qualifications
- Deep experience with high-performance networking protocols (InfiniBand, RoCE v2) and understanding of data movement physics.
- Fluent in C++ or Python with knowledge of NVIDIA architectures (H100/Blackwell) and memory hierarchy optimization.
- Comfort diving into TensorRT-LLM source code, writing custom C++/Python bindings, and debugging NVLink topology issues.
- Ability to discern when to use off-the-shelf solutions vs. building custom solutions due to upstream tooling limitations.
Skills
Required:High-performance networkingC++Python
Nice to have:NCCL/NVSHMEM
Benefits
- Competitive compensation, including meaningful equity
- (U.S. only) 100% coverage of medical, dental, and vision insurance for employee and dependents
- Flexible PTO policy including company wide Winter Break (office closed Dec 24–Jan 1)
- Paid parental leave
- Fertility and family-building stipend through Carrot
- (U.S. only) Company-facilitated 401(k)
- Exposure to a variety of ML startups, offering learning and networking opportunities.
Last updated Oct 21, 2026. Interested? Apply on the company site.
About Baseten
Inference is everything
Effortlessly serve optimized open source & custom models on the fastest, most reliable model delivery network.
AICloud InfrastructureDeveloper ToolsInference PlatformMachine LearningModel DeploymentModel Runtimesai-infrastructure-and-development
More open roles at Baseten
- Partner Enablement Lead4d agoHybrid (San Francisco), Hybrid (New York) · G&A
- Engineering Manager Runtime - Storage4d agoHybrid (San Francisco) · EPD
- Sales Manager - Enterprise (New York)1w agoHybrid (New York) · GTM
- GRC Manager1w agoHybrid (San Francisco), Hybrid (Remote) +4 more · EPD
- Software Engineer - Inference Platform2w agoHybrid (San Francisco), Hybrid (Toronto) +3 more · EPD