← All roles

Senior Network Production Engineer, AI Supercomputing

Microsoft· Software Engineering
Apply on company site
London£93.5k - £161.8kfull-timePosted 2w ago

About the role

Senior, hands-on Network Production Engineer to own the production health and operations of a multi-rail Ethernet backend network built on MRC and RDMA technologies for frontier-scale AI supercomputers. Focus on maximizing large-job success, minimizing performance degradation, and quick recovery from failures.

Responsibilities

  • Own Production Network Health – Own the availability, performance, and operational readiness of the MRC Ethernet backend network.
  • Define and operate service-level indicators for packet loss, congestion, link health, path diversity, bandwidth, tail latency, collective performance, and job impact.
  • Detect slow or degraded components before they cause training failures or reduce model FLOPs utilization.
  • Build health models that correlate switch, NIC, host, topology, and application telemetry.
  • Operate the Network at Frontier Scale – Participate in on-call rotation and lead the response to high-severity network incidents.
  • Diagnose failures spanning GPUs, NICs, switches, cables, firmware, drivers, network operating systems, MRC, NCCL, Kubernetes, and training workloads.
  • Develop automated mitigation mechanisms, including path avoidance, node quarantine, workload relocation, and safe component remediation.
  • Create operational procedures for maintenance, upgrades, rollback, capacity expansion, and topology changes.
  • Improve Large-Job Reliability and Performance – Work with training teams to investigate performance degradation, stalls, timeouts, job restarts, MFU loss.
  • Translate low-level network signals into clear job-level impact.
  • Establish network qualification gates for admitting nodes and racks into large training pools.
  • Run failure-injection and resilience testing for link, NIC, switch, and path failures.
  • Improve placement and routing for jobs spanning racks, rows, and network planes.
  • Automate Fleet Operations – Build production-quality software for validation, monitoring, diagnosis, remediation, and fleet-wide change management.
  • Replace manual investigation with deterministic workflows and actionable alerts.
  • Automate firmware, driver, configuration, and network OS compliance checks.
  • Maintain authoritative network inventory, topology, and configuration state.
  • Reduce mean time to detect, isolate, mitigate, and resolve failures.
  • Drive Cross-Company Execution – Lead technical investigations across internal infra teams, cloud/datacenter operators, Azure Networking, NVIDIA, switch/NIC teams, and partners.
  • Own incidents through resolution across organizational boundaries.
  • Produce clear root-cause analyses with actions, owners, and completion dates.
  • Translate recurring failures into requirements for future architectures.
  • Create runbooks and train engineers across regions for reliable round-the-clock coverage.
  • What Success Looks Like – Establish ownership and SLIs, build end-to-end visibility, reduce network-caused interruptions, automate detection/isolation, qualify/release new capacity, scalable operating model.
  • Primary success metrics – Large-scale training job success rate and goodput, network-attributed lost GPU hours, collective performance and tail latency, MTTR, detection/prevention metrics, automated remediation coverage, time from hardware availability to production-ready capacity.

Qualifications

  • Significant experience operating large-scale datacenter or high-performance computing networks in production.
  • Deep understanding of Ethernet, RDMA, RoCEv2, congestion control, routing, load balancing, and lossless or near-lossless network design.
  • Experience debugging complex failures across switches, NICs, hosts, drivers, firmware, and distributed applications.
  • Strong Linux systems knowledge and proficiency in Python, Go, C++, or another systems-oriented programming language.
  • Experience building monitoring, automation, diagnostic, or remediation systems for production infrastructure.
  • Demonstrated ability to lead high-severity incidents and coordinate resolution across multiple engineering teams.
  • Strong analytical skills and the ability to translate telemetry into a defensible root cause.
  • Willingness to participate in a global 24/7 on-call rotation.

Skills

Required:Network health monitoringMulti-rail Ethernet RDMA (MRC, RoCEv2)Incident managementAutomation and tooling for production
Nice to have:Telemetry and observability

Benefits

  • Find additional benefits and pay information here: https://careers.microsoft.com/v2/global/en/corporate-pay/united-kingdom-corporate-pay.html
Last updated Oct 9, 2026. Interested? Apply on the company site.

About Microsoft

Cloud ComputingConsumer ElectronicsSoftwareTechnologyliquid-ai-recruiting-target
Company profileWebsite

More open roles at Microsoft