AI Fabric Architect
fal.ai· Engineering
RemoteFull-timePosted yesterday
About the role
Owns the end-to-end network fabric architecture for fal’s GPU clusters, from design and vendor selection through hands-on site deployment, performance tuning, and incident response.
Responsibilities
- Own the fabric architecture for every site: GPU backend fabric (InfiniBand and/or RoCEv2), and GPU frontend Fabric
- Choose topologies and make the key calls: rail-optimized vs. other designs, InfiniBand vs. Ethernet, oversubscription ratios, and how the fabric scales from one pod to many
- Turn the architecture into a repeatable network blueprint: cabling plans, port maps, fabric addressing and routing design, and a fabric bill of materials (switches, NICs, optics, cables)
- Get hands-on with bring-up at each site: be on the floor for the first racks, validate cabling and firmware, configure the fabric manager and switches, and run performance acceptance (NCCL tests, burn-in) before capacity goes live
- Tune for real workload performance: congestion control, PFC/ECN, adaptive routing, and NCCL settings for both inference and training traffic
- Debug fabric problems wherever they show up, from a flapping link to a job that’s mysteriously slow
- Evaluate and select AI Fabric vendors and hardware, and push back with data when something doesn’t meet spec
- Collaborate across automation and software to help build network monitoring, telemetry, and automation that let a small team run a large fleet, and be the top escalation point for AI Fabric incidents
- Document as you go so each new site is faster than the last, and help hire and mentor the network engineers who join after you
Qualifications
- 7+ years in data center or HPC networking, including direct responsibility for production GPU cluster fabrics
- A track record of taking GPU fabrics from empty racks to production, with your own hands on the work, not only through vendors or integrators
- Deep expertise in InfiniBand (NDR/XDR, UFM, subnet management) and/or RoCEv2 lossless Ethernet
- Strong command of rail-optimized and Clos/fat-tree designs, BGP/ECMP, and EVPN/VXLAN
- A clear understanding of how fabric design and tuning affect collective communication and real workload performance
- Comfortable automating with Python, Ansible, or similar
- Strong Linux skills and the ability to debug across the boundary between host, NIC, and fabric
- Sound judgment on tradeoffs between cost, performance, reliability, and time to deploy
- NVIDIA Quantum/Spectrum-X, ConnectX-7/8, BlueField-3 SuperNICs, and/or Broadcom Tomahawk 5-class Ethernet
- 400G/800G OSFP optics, AOC/DAC, MPO trunking
Skills
Required:Data center or HPC networkingProduction GPU cluster fabric deploymentInfiniBandRoCEv2Network fabric designCollective communication performancePythonAnsibleLinuxTechnical tradeoff judgmentNVIDIA networking hardwareBroadcom Tomahawk EthernetHigh-speed optical and cabling
Nice to have:Neocloud or hyperscaler AI infrastructureNVLink / NVL72 systemsGPUDirect RDMA and GPUDirect StorageMulti-site deploymentsTravel
Benefits
- Interesting and challenging work
- Competitive salary and equity
- A lot of learning and growth opportunities
- We offer relocation assistance to San Francisco.
- Health, dental, and vision insurance (US)
- Regular team events and offsite
Last updated Nov 21, 2026. Interested? Apply on the company site.
About fal.ai
ai-infrastructure-and-developmentwing-list-2026wing-list-2026-late-stage
More open roles at fal.ai
- Compute SourcingyesterdaySan Francisco · Operations
- Legal OperationsyesterdaySan Francisco · Legal
- Senior Infrastructure Security Engineer2w agoSan Francisco · Engineering
- Enterprise Account Executive Manager, NAMER2w agoSan Francisco · Sales
- Staff Security Engineer2w agoSan Francisco · Engineering