← All roles

Site Reliability Engineer

xAI· Data Center
Apply on company site
Hybrid (Memphis/Southaven)full-timePosted 5d ago

About the role

Site Reliability Engineer focused on campus reliability for SpaceXSI, overseeing cross-discipline reliability across compute, network, storage, power, and cooling on a large AI training campus. Requires incident leadership, fleet-scale observability, and collaboration across software and facility boundaries.

Responsibilities

  • Own monitoring architecture and signal quality across GPU/host telemetry and facility telemetry; design alerting and suppress noise; treat alert noise as a design failure.
  • Provide SEV command support: incident leadership, bridge coordination with NOC, paging/escalation, and timeline/severity hygiene.
  • Run blameless postmortems and drive corrective actions to closure.
  • Gate high-risk changes: firmware/BIOS/BMC rollouts and SOP/MOP/EOP-governed work on power and cooling via disciplined change management.
  • Lead cross-functional reliability projects spanning compute, network, storage, and facility signal boundaries; address grey failures crossing boundaries.
  • Partner with Hardware Failure Analysis to turn fleet data into monitoring rules and vendor RCA inputs.
  • Build and maintain playbooks, run game days, and maintain cross-discipline dependency maps; own runbook quality with NOC.
  • Define error budgets and availability objectives at campus/service boundaries as adopted by the business.
  • Participate in on-call rotations and incident response for SEV-class events in Memphis/Southaven data center campus.

Qualifications

  • Bachelor's degree in Systems Engineering, Computer Science, Electrical Engineering, or related field (or equivalent experience).
  • 5+ years in site reliability, systems engineering, plant operations, or large-scale production operations (preferably in power plants, nuclear facilities, HPC, or data centers).
  • Proven large-scale incident command experience and calm technical leadership on a bridge (SRE/NOC bridge or plant control room).
  • Demonstrated monitoring/observability design at fleet/campus scale with alert hygiene and signal quality.
  • Hands-on experience with Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry and controls (SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus/BACnet).
  • Experience across at least two of: compute, network, storage, power, cooling / facilities telemetry.
  • Experience writing/operating playbooks, runbooks, or SOP/MOP/EOPs in 24/7 operations, control room, or NOC settings.
  • Proficiency in scripting (Python, Bash) for automation and analysis, plus familiarity with at least one systems language (C, C++, Java, Go, Rust, etc.).
  • Strong problem-solving with data-driven reliability focus.
  • Ability to collaborate with NOC, data center operations, hardware failure analysis, and infrastructure engineering.

Skills

Required:Monitoring and observability designIncident command and leadershipScripting (Python/Bash)Systems programming languagesChange management and SOPs
Last updated Oct 21, 2026. Interested? Apply on the company site.

About xAI

liquid-ai-recruiting-target
Company profileWebsite

More open roles at xAI