← All roles

Evaluations Engineer

Vals AI· Engineering & Research
Apply on company site
San Francisco$130k - $185kFull-timePosted 3mo ago

About the role

Engineers to own and run leaderboards evaluating LLMs against Vals AI benchmarks, testing new models, analyzing error modes, and reporting findings. Role involves collaboration with labs, startups, enterprises, and media to evaluate real-world task performance of foundation models.

Responsibilities

  • Evaluate new LLM model releases across the Vals AI suite of benchmarks
  • Work directly with both open-source and closed-source foundation model labs in evaluating model performance
  • Use tools like Docent to analyze common failure modes and patterns in model performance
  • Work directly with our social media team to post interesting findings and results
  • Add new models and maintain integrations in our model library
  • Help improve and maintain the infrastructure we use to run benchmarks (agentic and non-agentic)

Qualifications

  • Familiarity with the LLMs: You should already be familiar with the space - the current leading models, relative performance across them, how to use large language models in practice.
  • Strong engineering fundamentals: You can build and ship quickly with high quality. You should have a track record of building things of significant scope (at jobs, side projects, open source, etc.)
  • Python expertise: Significant experience in Python, especially in a professional setting.
  • Team collaboration: Experience working in development sprints, Git workflows, and pull request reviews.
  • Strong work ethic: Willingness to work long hours during model releases and get high-quality results out under tight deadlines.
  • Location: We are an in-person team based in San Francisco. We will support your relocation or transportation as needed.

Skills

Required:LLM familiarityPythonSprinting & Git workflows

Benefits

  • Highly competitive salary and meaningful ownership. Excellence is well rewarded.
  • Relocation and transportation support
  • Health/dental insurance coverage
  • Lunch and dinner provided, free snacks/coffee/drinks
  • 401K plan
  • Unlimited PTO
  • $1,500 housing stipend (within one-mile radius)
Last updated Oct 18, 2026. Interested? Apply on the company site.

About Vals AI

Private, domain-specific benchmarks in legal, tax, and finance.

financelawsoftwarehealthcarebenchmarksVals IndexSWE-benchTaxEval
Company profileWebsite

More open roles at Vals AI