← All roles

Member of Technical Staff - Pre-Training Evals Infra

Reflection· Research
Apply on company site
New York, NY, San Francisco, CA, London$0Full-timePosted 1w ago

About the role

Role focused on building evaluation infrastructure for pre-training models, enabling researchers to evaluate capabilities, guide training decisions, and assess release readiness. Involves evaluation authoring, distributed execution, results analysis, and researcher-facing tooling.

Responsibilities

  • Design and evolve evaluation abstractions. Build clear APIs and data models for evaluation tasks, datasets, configurations, model outputs, and scores, making it straightforward to add new evaluations and extend existing workflows.
  • Build scalable, reliable execution infrastructure. Develop distributed systems for evaluating model checkpoints throughout training and ahead of release, with robust orchestration, observability, and recovery from failures.
  • Optimize evaluation and inference performance. Profile inference servers and distributed execution to improve throughput, GPU utilization, and time to results.
  • Establish evaluation correctness and reproducibility. Build validation and regression tests, version datasets and configurations, and preserve the provenance needed to reproduce results and investigate discrepancies.
  • Build researcher-facing tools. Create intuitive interfaces and dashboards for inspecting model outputs, comparing checkpoints and experiments, and debugging unexpected behavior.
  • Operate and monitor evaluations as training progresses. Track evaluation workloads and results, triage failures, and develop alerts and diagnostics that help distinguish changes in model quality from inference or evaluation issues.
  • Turn research needs into reusable systems. Partner directly with researchers to identify bottlenecks, prioritize high-impact improvements, and turn bespoke workflows into self-service tools that work across teams.

Qualifications

  • Strong proficiency in Python, C++, or Rust, with the ability to reason deeply about concurrency, memory behavior, and performance.
  • Experience designing, building, and operating reliable, maintainable distributed systems, with sound judgment about interfaces, failure modes, and architectural tradeoffs.
  • Working knowledge of model inference and evaluation, including how implementation and configuration choices can affect measured model quality.
  • Ability to investigate complex problems across system boundaries, follow evidence, and identify root causes rather than just address symptoms.
  • Strong product judgment and experience translating the needs of researchers or other technical users into clear APIs and usable tools.
  • Comfort owning ambiguous problems end to end, prioritizing competing research needs, and balancing rapid delivery with long-term reliability and maintainability.
  • Attention to detail and the persistence to investigate subtle discrepancies in model outputs, evaluation results, and system behavior.

Skills

Required:PythonC++Distributed systems
Nice to have:Rust

Benefits

  • Top-tier compensation: Salary and equity structured to recognize and retain our talent globally.
  • Stock options: Everyone who joins and contributes to Reflection's success gets to share in the upside through stock options.
  • Health & wellness: Comprehensive medical, dental, vision, and life, with an annual wellness allowance.
  • Meals: Lunch and dinner are provided in the office daily.
  • Life & family: 22 weeks paid parental leave for all new birthing and non-birthing parents, including adoptive and surrogate journeys.
  • Vacation days: Unlimited paid time off in the U.S. and 30 days in the U.K.
  • Sponsorship support: We sponsor visas to help exceptional talent join our team and support long-term immigration pathways where applicable.
  • Team building: We have regular off-sites, happy hours, and team celebrations.
Last updated Oct 15, 2026. Interested? Apply on the company site.

About Reflection

Building frontier open intelligence

Building frontier open intelligence.

liquid-ai-recruiting-target
Company profileWebsite

More open roles at Reflection