Software Engineer, Inference Systems
Palo Alto, CA$200k - $420kFull-timePosted 1mo ago
About the role
Inference systems engineer focused on building engines that serve large models through the River API, optimizing fast, reliable inference with efficient GPU utilization. Role involves owning the serving runtime, batching, KV-cache management, distributed model execution, and checkpoint loading, supporting both customer-facing inference and RL sampling workloads.
Responsibilities
- Optimize inference for dense and mixture-of-experts models, including fine-tuned models and adapters.
- Improve batching, caching, and admission control to balance throughput, latency, memory use, and fairness.
- Accelerate multi-GPU execution, communication, and model loading while preserving model-version consistency.
- Improve RL sampling throughput while keeping samples and log probabilities tied to the correct model version.
- Build reliable streaming, cancellation, and recovery under failures and overload.
- Profile bottlenecks and validate improvements through reproducible performance and correctness tests.
Qualifications
- Bachelor’s degree in Computer Science, Computer Engineering, or equivalent practical experience.
- Experience building inference engines or performance-sensitive distributed services.
- Strong understanding of transformer inference, GPU memory, concurrency, and networking.
- Proficiency in Python and C++ or Rust.
- Strong debugging and profiling skills across models, runtimes, and services.
- A collaborative mindset and strong ownership of engineering outcomes.
Skills
Required:Inference engine developmentTransformer inferencePythonC++ or RustDebugging and profiling
Benefits
- Comprehensive health, dental, and vision insurance; unlimited PTO; relocation assistance as needed.
Last updated Sep 23, 2026. Interested? Apply on the company site.
About River AI
Intelligence that flows with you
River is building a new stack for Personal AI.
LoRA fine-tuningReinforcement learningOpen modelsPay per tokenAPIPersonal AI
More open roles at River AI
- IT Lead2w agoPalo Alto, CA · Staff
- Design Verification Emulation, Hardware3w agoHybrid (Austin, TX or Palo Alto, CA) · Personal AI Hardware
- People Lead4w agoPalo Alto, CA · Staff
- Operations Lead4w agoPalo Alto, CA · Staff
- Software Engineer, GPU Kernels1mo agoPalo Alto, CA · River API