Member of Technical Staff — Training Infrastructure
Causal Labs· Infrastructure
San Francisco, SingaporeFull-timePosted 2mo ago
About the role
Infra role focused on designing, implementing, and optimizing large-scale distributed training systems for a Large Physics foundation Model (LPM), enabling fast, efficient, and reliable training across thousands of GPUs.
Responsibilities
- Design, implement, and optimize distributed training systems that scale across thousands of GPUs
- Research and test parallelization strategies and numerical precision trade-offs across model scales, including architectures that don’t map cleanly onto existing LLM training stacks
- Analyze, profile, and debug low-level GPU operations to maximize throughput and hardware utilization
- Build reusable frameworks for checkpointing, fault tolerance, and reproducibility that stay robust under rapid research iteration
- Collaborate with researchers to bring novel model architectures from prototype to full scale
- Stay up-to-date on research to bring new ideas to work
Qualifications
- Demonstrated proficiency with distributed training frameworks and techniques (e.g. FSDP, DeepSpeed, Megatron, Pytorch, JAX/XLA) to train large foundation models
- Strong grasp of state-of-the-art techniques for optimizing training workloads: parallelism strategies, memory optimization, mixed precision, communication overlap
- Ability to profile and debug performance in complex codebases, from framework internals down to kernels and collectives
- Deep understanding of deep learning frameworks (e.g. PyTorch, JAX) and their underlying system architectures
- Bonus: contributions to open-source ML infrastructure (e.g. PyTorch, Megatron-LM, DeepSpeed, XLA)
Skills
Required:Distributed training frameworksParallelism strategiesMemory optimizationMixed precisionProfiling and debugging
Last updated Oct 2, 2026. Interested? Apply on the company site.
About Causal Labs
Towards general causal intelligence
Causal Labs is building a Large Physics foundation Model (LPM) to learn causality through physics and weather
causalityphysicsweatherAIfoundation model
More open roles at Causal Labs
- Member of Technical Staff — DevOps3w agoSan Francisco, Singapore · Infrastructure
- Executive Assistant & Office Manager2mo agoSan Francisco, Singapore · Strategy & Operations
- Member of Technical Staff — Research, Atmospheric Science2mo agoSan Francisco, Singapore · Research
- Member of Technical Staff — Research, Physics2mo agoSan Francisco, Singapore · Research
- Member of Technical Staff — ML Research, Planning2mo agoSan Francisco, Singapore · Research