← All roles

Staff Software Engineer, Inference API

Cerebras· Software Engineering
Apply on company site
Hybrid (Sunnyvale, CA), Hybrid (Toronto, CAN)Full-timePosted 4w ago

About the role

Role focused on building and evolving the ML API layer for a disaggregated AI inference system, enabling production-ready, API-first access to multiple backend accelerators and model types.

Responsibilities

  • Build production ML inference APIs. Design, implement, and maintain APIs for chat completions, text generation, streaming, model configuration, tool calling, structured outputs, multimodal inputs, and other emerging inference capabilities.
  • Deliver a unified serving experience. Create consistent request and response semantics across GPU prefill, Cerebras decode, and other heterogeneous inference backends.
  • Enable new models and capabilities. Integrate emerging foundation models, tokenizers, prompt formats, sampling methods, attention variants, multimodal inputs, and model-specific features into the serving platform.
  • Own API compatibility and evolution. Maintain compatibility with widely adopted inference interfaces while designing Cerebras-specific extensions. Establish clear versioning, deprecation, validation, and backward compatibility practices.
  • Integrate with model-serving runtimes. Extend and integrate custom inference services with vLLM, PyTorch, Hugging Face libraries, the AMD ROCm stack, and Cerebras runtime components.
  • Support disaggregated inference. Build the control and data paths required to coordinate GPU prefill with Cerebras decode, including request routing, state transfer, error handling, retries, and lifecycle management.
  • Improve serving performance. Optimize streaming behavior, time to first token, request latency, throughput, batching, serialization, tokenization, scheduling, and communication between serving components.
  • Ensure functional and numerical correctness. Build validation systems for tokenization, sampling, logits, generated outputs, precision changes, model upgrades, determinism, and compatibility across serving backends.
  • Strengthen reliability and observability. Define end-to-end service indicators and build structured logging, tracing, metrics, dashboards, health checks, and diagnostic tooling for production inference traffic.
  • Develop testing and qualification infrastructure. Create conformance tests, workload-replay tools, model-validation suites, performance benchmarks, integration tests, and release gates.
  • Improve developer experience. Build intuitive configuration, SDKs, documentation, examples, debugging tools, and self-service workflows for internal developers, customers, and partners.
  • Collaborate across the stack. Partner with compiler, runtime, kernel, cloud, product, and solutions teams to translate model and customer requirements into scalable serving capabilities.

Qualifications

  • 5+ years of software engineering experience, including substantial individual-contributor ownership of production software or distributed systems.
  • Strong programming ability in Python and Go plus experience developing performance-sensitive or highly concurrent services in C++, Rust, or a similar systems language.
  • Experience building stable APIs with clear validation, error handling, observability, compatibility, and versioning practices.
  • Experience integrating software across service, framework, runtime, and infrastructure boundaries.
  • Experience designing or maintaining OpenAI-compatible, gRPC, REST, or streaming inference APIs.
  • Experience with Linux, containers, Kubernetes or comparable orchestration systems, CI/CD, and operating latency-sensitive services in production.
  • Ability to diagnose correctness, reliability, and performance issues across multiple components of a distributed serving system.
  • Strong communication and cross-functional execution skills, with the ability to turn ambiguous model or product requirements into production-quality software.
  • Bachelor's degree in computer science, Computer Engineering, Electrical Engineering, or a related discipline, or equivalent practical experience.

Skills

Required:PythonGo
Nice to have:C++/Rust
Last updated Oct 18, 2026. Interested? Apply on the company site.

About Cerebras

Cerebras is the go-to platform for fast and effortless AI training. Learn more at cerebras.ai.

Cerebras is the go-to platform for fast and effortless AI training.

AICerebras Wafer-Scale Enginechipshardwareinferenceliquid-ai-recruiting-targetmultimodalsetter-30
Company profileWebsite

More open roles at Cerebras