Engineering Lead, Inference Optimization
Venice AI· Engineering
full-timePosted 5d ago
About the role
Lead the technical strategy and team for inference optimization at Venice AI, driving peak inference performance at scale while shaping overall architecture and hardware strategy.
Responsibilities
- Own Venice’s technical strategy for inference performance
- Recruit and lead the Inference Optimization Team at Venice
- Optimize Venice's GPU infrastructure across architectures (e.g. H200s, B300s)
- Improve latency, throughput, and cost per token for LLM inference workloads
- Build reproducible benchmarking harnesses across inference engines (e.g. vLLM, SGLang) to identify the optimal engine, quantization scheme, and parallelism strategy per workload and GPU SKU
- Work with the inference routing system to optimize multivariate inference load-balancing algorithms
- Evaluate emerging inference optimization techniques (custom CUDA/Triton kernels), novel attention variants, new quantization schemes, and compilation stack improvements
- Evaluate emerging inference hardware (FPGAs, ASICs, custom silicon) for viability in Venice's stack
Qualifications
- 8+ years in performance optimization or HPC, with deep GPU architecture and parallel programming knowledge
- 5+ years experience leading engineering teams
- Proficiency in Python, Rust, or Go
- Bonus: C++/CUDA
- Hands-on experience with at least one production LLM inference engine (e.g. vLLM, SGLang) running at high volume in production
- Demonstrated experience with LLM inference optimization techniques: continuous batching, PagedAttention/KV cache management, speculative decoding, quantization, CUDA graphs, and torch.compile
- Fluency with quantization tradeoffs, both qualitative and quantitative
- Experience with distributed inference strategies (tensor parallelism, pipeline parallelism, MoE parallelism) in multi-GPU and multi-node environments
- Fluency with GPU profiling (Nsight Systems, Nsight Compute, PyTorch Profiler) and a bias toward measuring before optimizing
- Bonus: diffusion/image model inference optimization, custom Triton kernels, contributions to open-source inference frameworks
Last updated Oct 22, 2026. Interested? Apply on the company site.
About Venice AI
More open roles at Venice AI
- Senior Data Scientist, Business Analytics5d agoRemote · Business Operations
- Staff Data Engineer5d agoRemote · Engineering
- Senior Data Scientist, Business Analytics5d agoRemote · Business Operations
- Staff Product Engineer, Gen AI5d agoRemote · Engineering
- Product Manager, Models1w agoRemote · Product