Member of Technical Staff - Research Engineer
Hybrid (SF / Freiburg)$180k - $290k + equityfull-timePosted 3mo ago
About the role
Roles focused on production training for large-scale multimodal generative models, combining research and engineering to stabilize, optimize, and scale training systems across GPU fleets.
Responsibilities
- Improve the performance, reliability, and numerical stability of production training runs for large multimodal generative models
- Profile full training steps across model code, attention, kernels, data loading, encoders, communication, optimizer steps, checkpointing, and memory pressure
- Implement and validate GPU-level optimizations: fused kernels, attention paths, low-precision matmuls, quantization kernels, CUDA/Triton/CuTe/CUTLASS experiments, and no-compile alternatives where they make sense
- Push lower-precision training forward, including FP8 / MXFP8 / FP4-style paths, weight and activation quantization, accumulation choices, convergence risk, and quality tradeoffs against baseline training runs
- Work with researchers to translate architecture changes into efficient training implementations, and help distinguish real model-quality progress from changes that only look good in a microbenchmark
- Debug distributed training failures: NaNs, loss spikes, silent numerical drift, memory leaks, stragglers, bad nodes, NCCL issues, and throughput cliffs
- Build benchmarking and profiling harnesses that make performance claims trustworthy across hardware, shapes, sequence lengths, and training configurations
- Help the training team move quickly when an urgent bottleneck appears, while turning repeated failures into better abstractions and tools
Qualifications
- Experience working deeply on large-scale training systems, ideally as part of a training group working closely with researchers
- Strong PyTorch fluency, including comfort reading and modifying low-level training code rather than only using high-level APIs
- Experience with distributed training concepts such as FSDP, tensor/model/context/sequence parallelism, activation checkpointing, NCCL, and overlapping compute and communication
- Hands-on experience improving training throughput, memory footprint, or stability in real training runs
- Experience profiling GPU workloads with tools like Nsight Systems, Nsight Compute, torch profiler, trace viewers, or custom telemetry
- Practical GPU performance judgment: ability to verify correctness, numerical behavior, and performance, and own the result
- Understanding of low-precision training and quantization tradeoffs: FP8, MXFP8, FP4/NVFP4-style formats, scaling, accumulation, numerical validation, and convergence risk
- Good research judgment: ability to partner with researchers on ablations and tie optimization to model-quality outcomes
- Comfortable operating in ambiguity (implementation, production issues, or evaluating explanations)
Skills
Required:PyTorchDistributed trainingProfiling GPU workloadsLow-precision training and quantizationResearch collaboration
Nice to have:Kernel development / GPU programming
Last updated Jul 29, 2026. Interested? Apply on the company site.
About Black Forest Labs
The frontier AI research lab for visual intelligence.
Cutting-edge generative AI models from Black Forest Labs.
APIenterprise AIgenerative AIimage generationliquid-ai-recruiting-targetmodel deploymentopen weightsphoto-realistic output
More open roles at Black Forest Labs
- Senior Partnerships Manager2mo agoHybrid (San Francisco / US locations) · GTM
- Field Marketing Manager2mo agoMarketing
- IT Engineer2mo agoFreiburg, Germany, Hybrid (San Francisco, USA) · Engineering
- Product Engineer2mo agoHybrid (Freiburg, Germany) · Product
- Management Accountant3mo agoFreiburg · Finance