← All roles

Staff GPU Inference SDET

Cerebras· Software Engineering
Apply on company site
Hybrid (Sunnyvale, CA), Hybrid (Toronto, CAN)Full-timePosted 1mo ago

About the role

Founding quality, reliability, and validation lead for a new GPU Inference Development team. Responsible for designing, building, and scaling end-to-end release qualification and automated test ecosystem for GPU inference stack and rack-scale accelerated compute fleets, including multi-node GPU cluster bring-up, prefill optimizations, serving engines, and ensuring numerical correctness and performance under real-world workloads.

Responsibilities

  • Build GPU Release Qualification Systems: Design and implement automated test automation frameworks, regression gates, and release qualification pipelines for the complete GPU inference stack—spanning custom API services, model-serving workers, container runtimes, serving engines, driver stacks, and firmware.
  • Inference Serving & Workload Validation: Benchmark and stress-test distributed LLM serving frameworks, focusing on prefill vs. decode worker performance, continuous batching, prefix caching, KV-cache efficiency, and tensor/expert parallelism.
  • Performance & Performance Modeling Verification: Build automated workload replay and benchmarking tools to validate GPU performance models. Track critical serving metrics including Time-to-First-Token (TTFT), Inter-Token Latency (ITL), request throughput, tail latency (P99), and capacity efficiency.
  • Numerical Correctness & Quality Gates: Build validation infrastructure to ensure model accuracy, precision stability (FP16/FP8/quantization), determinism, and output correctness across software updates, kernel fusions, and hardware revisions.
  • Fault Injection & Fleet Resilience: Engineer chaos engineering and fault-injection suites to simulate node failures, inter-node network degradation, GPU memory leaks, driver/firmware mismatches, and automated recovery paths for multi-node GPU clusters.
  • Observability & CI/CD Integration: Integrate automated test pipelines with telemetry tools (e.g., Prometheus, Grafana) to turn one-off investigations into repeatable engineering gates and continuous performance monitoring.

Qualifications

  • 8+ years of software engineering experience as an SDET, Infrastructure Quality Lead, or Systems Test Engineer.
  • Hands-on experience bringing up, provisioning, and validating multi-node GPU clusters (NVIDIA or AMD ecosystem) across public cloud infrastructure or enterprise data center environments.
  • Deep understanding of LLM serving engines and distributed runtimes, including prefill vs. decode disaggregation, KV-cache management, and dynamic batching.
  • Expert-level Python programming skills with extensive experience designing custom test automation frameworks, diagnostic tooling, and CI/CD integration.
  • Strong proficiency with container orchestration tools (e.g., Kubernetes, Slurm, Ray) and high-performance cluster interconnects (e.g., InfiniBand, RoCE, NCCL).
  • Proven background in root-cause analysis across software/hardware boundaries, stress testing, and node failure simulation in distributed systems.

Skills

Required:PythonGPU cluster managementLLM serving enginesContainer orchestrationNetworking & interconnects

Benefits

  • Not specified in description
Last updated Sep 13, 2026. Interested? Apply on the company site.

About Cerebras

Cerebras is the go-to platform for fast and effortless AI training. Learn more at cerebras.ai.

Cerebras is the go-to platform for fast and effortless AI training.

AICerebras Wafer-Scale Enginechipshardwareinferenceliquid-ai-recruiting-targetmultimodalsetter-30
Company profileWebsite

More open roles at Cerebras