← All roles

GPU Cluster Engineer, Networking

Sciforium· Engineering
Apply on company site
San Francisco, CA$150k - $180kFull-timePosted 2mo ago

About the role

Senior Network Engineer responsible for end-to-end networking stack of GPU clusters, including RDMA compute fabric, data center perimeter, cross-site and cloud connectivity, and production operations.

Responsibilities

  • Fabric Architecture & Cluster Network Design - Greenfield Design: Architect full network for new GPU clusters — compute/backend fabric, storage network, in-band frontend, out-of-band management (BMC/IPMI), and perimeter — including topology (fat-tree/Clos, rail-optimized), oversubscription analysis, and growth planning. - RDMA Fabric Selection: Design and justify InfiniBand (NDR/XDR, Quantum-class) vs. lossless Ethernet/RoCE v2 (Spectrum-X, Tomahawk-class) fabrics for training and inference workloads, including scale-up domains (NVLink/NVSwitch, NVL72-class rack systems). - Hardware & Physical Plant: Specify switches, NICs (ConnectX/BlueField), optics (400/800G OSFP/QSFP-DD), and cabling plans; produce the port maps and cable schedules. - Logical Design: Own IP addressing plans, VLAN/VRF segmentation, and BGP/EVPN-VXLAN underlay-overlay architecture. - RDMA & Performance Engineering - Lossless Ethernet: Configure and validate RoCE v2 at scale — PFC, ECN/DCQCN congestion control, QoS/DSCP classification, buffer tuning — and prove no-drop behavior under full training load. - InfiniBand Operations: Run subnet managers (UFM/OpenSM), adaptive routing, SHARP in-network aggregation, partition keys, and virtual-lane QoS. - Fabric Validation: Benchmark and sign off new fabrics with perftest/ib_write_bw and nccl-tests/rccl-tests; verify GPUDirect RDMA paths and topology-aware placement. - Performance Debugging: Root-cause fabric-level issues — congestion trees, ECMP polarization, link flaps, degraded optics, straggler nodes caused by the network. - Production Network Operations & Security - Routing & Switching: Configure and maintain BGP, OSPF, ECMP, and EVPN-VXLAN across multi-vendor NOSes (Arista EOS, NVIDIA Cumulus, Cisco NX-OS, Junos, SONiC). - Perimeter & Security: Own firewalls, VPNs (WireGuard/IPsec), NAT, ACLs, and network segmentation between production, research, and management domains. - Network as Code: Generate configs from a source of truth (NetBox), deploy via Ansible/Nornir/NAPALM with Git-reviewed, CI-validated changes and zero-touch provisioning for new switches. - Telemetry: Build fabric observability — streaming telemetry/gnmi, sFlow, optics DOM, buffer/drop counters — into Prometheus/Grafana with actionable alerting. - Lifecycle & On-Call: Run switch/NIC firmware upgrade campaigns with minimal workload disruption, and escalation point for network incidents. - Cross-Cluster & Cloud Connectivity - Inter-Site Links: Design and operate dark fiber and DWDM capacity between sites, including latency budgeting and carrier/IX peering. - Hybrid Cloud: Architect AWS Direct Connect / Azure ExpressRoute / GCP Interconnect links, VPC and Transit Gateway designs, and VPN failover paths. - Data Movement: Engineer WAN QoS and traffic paths for cross-cluster replication, dataset transfer, and checkpoint traffic.

Qualifications

  • 7+ years designing and operating production data center networks, including at least one large-scale HPC/AI cluster fabric (InfiniBand or RoCE v2) you designed or ran.
  • Expert-level routing and switching: BGP (including EVPN-VXLAN), OSPF, ECMP, VRF segmentation, across at least two major vendor platforms.
  • Deep RDMA expertise: lossless RoCE v2 tuning (PFC/ECN/DCQCN) and/or InfiniBand fabric management (subnet managers, adaptive routing, SHARP).
  • Hands-on experience with 100–800G optics, high-radix switching, and Clos/rail-optimized topologies.
  • Production network security experience: enterprise firewalls, site-to-site and remote-access VPNs, segmentation design.
  • Network automation proficiency: Python plus Ansible (or Nornir/NAPALM), with Git-based configuration workflows.
  • Working knowledge of the host-side RDMA stack (MOFED/DOCA, NIC tuning, GPUDirect) and how NCCL/RCCL collectives map onto physical fabric.
  • Cloud networking experience: VPC design and dedicated interconnects (Direct Connect/ExpressRoute class).

Skills

Required:Network design and architectureBGP/EVPN-VXLANRDMA (RoCE v2 / InfiniBand)Network automation

Benefits

  • Medical, dental, and vision insurance
  • 401k plan
  • Daily lunch, snacks, and beverages
  • Flexible time off
  • Competitive salary and equity
Last updated Sep 25, 2026. Interested? Apply on the company site.

About Sciforium

Byte-Native AI Models & Infrastructure

Sciforium is pioneering the future of AI infrastructure and research. We're developing byte-native multimodal foundation models while delivering serverless LLM serving at a fraction of traditional costs

Company profileWebsite

More open roles at Sciforium