← All roles

Research Engineer - Evals

AGI, Inc.· Engineering
Apply on company site
San Francisco OfficeFull-timePosted 2mo ago

About the role

Build and govern the evaluation harnesses and dashboards to measure model capability, agentic behavior, on-device performance, and end-user experience. Define ship-ready criteria and protect against deadline-driven compromises.

Responsibilities

  • The eval suites that gate every model and agent release — capability, behavior, regressions, and human-rated rubrics that catch what automated evals miss
  • The dashboards and tooling that make researcher experiment loops fast and leadership decisions easy
  • The bar — what counts as ready to ship, and how we know
  • Research by ensuring what we measure is what we want
  • Instrument real-user behavior on real devices
  • Translate “did it get better” into language an OEM partner can hold us to

Skills

Required:Measuring non-deterministic systems (agent eval, tool use, long-horizon tasks, multilingual behavior)Pushing back on metrics that are gamed

Benefits

  • Top-tier relocation and immigration support
  • In person in SF
Last updated Aug 7, 2026. Interested? Apply on the company site.

About AGI, Inc.

We are an AI lab bringing superintelligence to the edge, on a mission to make it 100% secure, private and accessible.

AIsecurityprivacyaccessibilitydemocareersblogmerch
Company profileWebsite

More open roles at AGI, Inc.