Software Engineer - Training/Inference (C++)
xAI· Infrastructure
Hybrid (Palo Alto, CA), Hybrid (Seattle, WA)$180k - $440kfull-timePosted 11mo ago
About the role
Role focusing on building and optimizing a high-performance inference platform for Grok, serving billions of users with low latency and high reliability. End-to-end ownership of distributed infrastructure and low-level optimizations, including GPU kernels and model-serving internals.
Responsibilities
- Architect and implement scalable distributed infrastructure for model serving (load balancing, auto-scaling, batch scheduling, global KV cache).
- Optimize latency and throughput of model inference under real production workloads.
- Build reliable, high-concurrency serving systems that serve billions of users with 100% uptime, 0% error rate, and excellent tail latency.
- Benchmark, fine-tune, and accelerate inference engines (including low-level GPU kernel work and code generation).
- Develop custom tools to trace, replay, and fix issues across the full stack — from orchestration down to GPU kernels.
- Create robust CI/CD infrastructure for seamless endpoint deployment, image publishing, and inference engine updates.
- Accelerate research on scaling test-time compute, RL rollout, and model-hardware co-design for next-generation systems.
Qualifications
- Deep low-level systems programming (C/C++ or Rust)
- Experience with large-scale, high-concurrent production serving.
- Experience with GPU inference engines (vLLM, SGLang, Triton, TensorRT-LLM, etc.).
- Strong background in system optimizations: batching, caching, load balancing, parallelism.
- Low-level inference optimizations: GPU kernels, code generation.
- Algorithmic inference optimizations: quantization, speculative decoding, distillation, low-precision numerics.
- Experience with testing, benchmarking, and reliability of inference services.
- Experience designing and implementing CI/CD infrastructure for inference.
Skills
Required:C/C++ (or Rust)Large-scale, high-concurrency production servingGPU inference engines
Benefits
- equity
- comprehensive medical, vision, and dental coverage
- access to a 401(k) retirement plan
- short & long-term disability insurance
- life insurance
- various other discounts and perks
Last updated Oct 14, 2026. Interested? Apply on the company site.
More open roles at xAI
- Supervisor, Child Safety Operations4d agoAustin, TX · Safety
- Security Engineer, Kubernetes4d agoHybrid (Palo Alto, CA; Austin, TX; New York, NY; Washington, DC) · Information Security
- Security Engineer - GRC Sales4d agoInformation Security
- Site Reliability Engineer4d agoHybrid (Memphis/Southaven) · Data Center
- Senior Linux Data Center Technician - Memphis4d agoHybrid (Memphis, TN) · Data Center