← All roles

Research Scientist, APEX Benchmarks

Mercor· Research
Apply on company site
San Francisco$200k - $500k + equityFull-timePosted 1mo ago

About the role

Lead the design of the next generation of APEX benchmarks and the expert-built datasets behind them, shaping how frontier models are evaluated and how expert-curated data translates into business value.

Responsibilities

  • Benchmark design: Decide what the next APEX benchmark should measure, based on frontier model performance, saturation of existing evals, and where economically valuable work is still out of reach. Own the task taxonomy, difficulty calibration, contamination controls and statistical design.
  • Dataset design: Design expert-built datasets and grading rubrics at scale — deciding what makes a task hard, what makes a grade defensible, and how to hold quality while thousands of experts produce work in parallel.
  • Measurement rigor: Set the standard for how we report results: confidence intervals, inter-rater agreement, human vs. model-as-judge calibration, held-out splits, and the failure analysis that explains why a model scored the way it did.
  • Partnerships: Work with academic collaborators and industry partners to co-design benchmarks and get them adopted.
  • External research voice: Publish — arXiv papers, open datasets, blog posts, conference talks, leaderboard releases — and represent Mercor’s research in conversations with frontier labs, customers and the press.
  • Translate results into narrative: Turn benchmark findings into clear arguments about the ROI of expert-curated data for reports and GTM materials.
  • Cross-functional work: Partner with data operations, engineering, product and strategy to take a benchmark from design to production, and surface research findings that shape the roadmap.
  • Stay at the frontier: Track LLM evaluation literature and bring good practices into Mercor’s benchmark construction.

Qualifications

  • Research background in evaluation: Strong applied or academic research background in LLM evaluation, benchmarking, NLP or related field, with rigorous experimental design.
  • Judgment about what to measure: Ability to identify informative measurements rather than easiest-to-build ones.
  • Statistical rigor: Ability to reason about sampling, variance, contamination and grader reliability; defendable results.
  • Hands-on: Strong coding skills; able to build an evaluation harness, run experiments and analyze results.
  • Exceptional communication: Able to present complex findings clearly to researchers and non-technical audiences.
  • Comfort with ambiguity: Experience working in fast-moving, cross-functional environments with undefined problem spaces.
  • Interest in the commercial side: Curiosity about GTM strategy, startup dynamics and AI data business.
  • In-person: Willing to work from San Francisco office five days a week.

Skills

Required:Evaluation design and methodologyExperimental coding and tooling

Benefits

  • Bi-annual performance bonus structure
  • Generous equity grant vested over 4 years
  • Up to $15k Relocation bonus
  • $10K housing bonus (if you live within 0.5 miles of our office)
  • $1.5K monthly stipend for meals
  • Free Equinox membership
  • $200 monthly laundry reimbursement
  • $200 monthly personal wellness reimbursement
  • Health, Dental, Vision insurance
  • 401(k) with company match
Last updated Sep 9, 2026. Interested? Apply on the company site.

About Mercor

Shape the future of work

Find top-tier, remote, AI roles for your expertise. Available only on Mercor.

AIremotejobstechnology
Company profileWebsite

More open roles at Mercor