← All roles

Member of Technical Staff, Enterprise Evals Platform

Mercor· Engineering
Apply on company site
San Francisco$220k - $425kFull-timePosted 2mo ago

About the role

Enterprise platform engineering role focused on building evaluation systems for enterprise agents, including verifiers, environments, and grading infrastructure to enable scalable, measurable evaluation of agent runtimes and prompts across domains and tasks.

Responsibilities

  • Define golden sets: decompose real tasks and encode the expert quality bar.
  • Build verifiers over agent trajectories and outputs, calibrated and hard to game.
  • Build the eval platform that runs offline environments, task suites, and grading at scale.
  • Run loss analysis over production trajectories and turn failure modes into regression tests.
  • Run the optimization loop across models, prompts, skills, and harnesses.
  • Own the rollout gates that decide whether an agent change ships.
  • Partner with the Enterprise Platform team and the Applied AI engineers embedded with customers.

Qualifications

  • Professional, academic, or research experience in agent engineering and evaluation, including how agent runtimes and harnesses produce a trajectory and where it fails.
  • Bachelor’s Degree in Computer Science or a related technical degree such as Electrical Engineering, Physics, Mathematics, Computer Engineering, Information Systems or Statistics
  • 1–5 years in industry level software engineering
  • Experience building evaluation suites for LLM or agent systems, and familiarity with how benchmarks such as terminal-bench, tau-bench, and APEX are constructed and where they get gamed.
  • Judgment about task and rubric design: turning a fuzzy notion of quality into something measurable, with agent or model improvements to show for it.
  • Strong software engineering fundamentals, and the ability to work independently on ambiguous, loosely specified problems.
  • Bonus: experience with Harbor environments and RL environments.

Skills

Required:Agent engineering and evaluationEvaluation suites (LLM/agent systems)

Benefits

  • Up to $15k relocation bonus
  • $10K housing bonus (if you live within 0.5 miles of our office)
  • $1.5K monthly stipend for meals
  • Generous equity grant vested over 4 years
  • Free Equinox membership
  • $200 monthly laundry reimbursement
  • $200 monthly personal wellness reimbursement
  • Health, Dental, Vision insurance
Last updated Oct 14, 2026. Interested? Apply on the company site.

About Mercor

Shape the future of work

Find top-tier, remote, AI roles for your expertise. Available only on Mercor.

AIremotejobstechnology
Company profileWebsite

More open roles at Mercor