← All roles

AI Fabric Architect

fal.ai· Engineering
Apply on company site
RemoteFull-timePosted yesterday

About the role

Owns the end-to-end network fabric architecture for fal’s GPU clusters, from design and vendor selection through hands-on site deployment, performance tuning, and incident response.

Responsibilities

  • Own the fabric architecture for every site: GPU backend fabric (InfiniBand and/or RoCEv2), and GPU frontend Fabric
  • Choose topologies and make the key calls: rail-optimized vs. other designs, InfiniBand vs. Ethernet, oversubscription ratios, and how the fabric scales from one pod to many
  • Turn the architecture into a repeatable network blueprint: cabling plans, port maps, fabric addressing and routing design, and a fabric bill of materials (switches, NICs, optics, cables)
  • Get hands-on with bring-up at each site: be on the floor for the first racks, validate cabling and firmware, configure the fabric manager and switches, and run performance acceptance (NCCL tests, burn-in) before capacity goes live
  • Tune for real workload performance: congestion control, PFC/ECN, adaptive routing, and NCCL settings for both inference and training traffic
  • Debug fabric problems wherever they show up, from a flapping link to a job that’s mysteriously slow
  • Evaluate and select AI Fabric vendors and hardware, and push back with data when something doesn’t meet spec
  • Collaborate across automation and software to help build network monitoring, telemetry, and automation that let a small team run a large fleet, and be the top escalation point for AI Fabric incidents
  • Document as you go so each new site is faster than the last, and help hire and mentor the network engineers who join after you

Qualifications

  • 7+ years in data center or HPC networking, including direct responsibility for production GPU cluster fabrics
  • A track record of taking GPU fabrics from empty racks to production, with your own hands on the work, not only through vendors or integrators
  • Deep expertise in InfiniBand (NDR/XDR, UFM, subnet management) and/or RoCEv2 lossless Ethernet
  • Strong command of rail-optimized and Clos/fat-tree designs, BGP/ECMP, and EVPN/VXLAN
  • A clear understanding of how fabric design and tuning affect collective communication and real workload performance
  • Comfortable automating with Python, Ansible, or similar
  • Strong Linux skills and the ability to debug across the boundary between host, NIC, and fabric
  • Sound judgment on tradeoffs between cost, performance, reliability, and time to deploy
  • NVIDIA Quantum/Spectrum-X, ConnectX-7/8, BlueField-3 SuperNICs, and/or Broadcom Tomahawk 5-class Ethernet
  • 400G/800G OSFP optics, AOC/DAC, MPO trunking

Skills

Required:Data center or HPC networkingProduction GPU cluster fabric deploymentInfiniBandRoCEv2Network fabric designCollective communication performancePythonAnsibleLinuxTechnical tradeoff judgmentNVIDIA networking hardwareBroadcom Tomahawk EthernetHigh-speed optical and cabling
Nice to have:Neocloud or hyperscaler AI infrastructureNVLink / NVL72 systemsGPUDirect RDMA and GPUDirect StorageMulti-site deploymentsTravel

Benefits

  • Interesting and challenging work
  • Competitive salary and equity
  • A lot of learning and growth opportunities
  • We offer relocation assistance to San Francisco.
  • Health, dental, and vision insurance (US)
  • Regular team events and offsite
Last updated Nov 21, 2026. Interested? Apply on the company site.

About fal.ai

ai-infrastructure-and-developmentwing-list-2026wing-list-2026-late-stage

More open roles at fal.ai