← All roles

Member of Technical Staff (Language Model Evaluations)

Artificial Analysis· Technical Staff (incl. Product)
Apply on company site
San Francisco (On-site)Full-timePosted 1mo ago

About the role

Role focused on designing and running frontier language model evaluations, building datasets and infrastructure, publishing analyses, and benchmarking pre-release models, with on-site in San Francisco.

Responsibilities

  • Design Next-Generation Frontier Evals: Conceive and ship the next generation of frontier evaluations, like AA-Briefcase and AA-Omniscience, across reasoning, knowledge, coding, agentic capability and beyond
  • Build Evaluation Datasets and Infrastructure: Construct the datasets, harnesses and scoring systems behind our benchmarks, engineered for contamination resistance and repeatability at frontier scale
  • Shape the Future Intelligence Index: The evaluations you build will contribute to future versions of the Artificial Analysis Intelligence Index and other areas of our platform, defining how the industry measures frontier capability
  • Publish Influential Analysis: Produce the reports, indexes and data visualizations that shape how the industry understands language model progress
  • Work with Frontier Labs on Pre-Release Models: Benchmark the leading labs’ systems, including pre-release and newly launched models, working directly with their research teams; our commercial team owns client relationships day to day, so your time stays on the science
  • Evaluate Every Major Model: Run our evaluation suite across frontier releases as they land, and own the integrity of the results the industry quotes
  • Become AI-Native: Embrace an AI-native workflow, using cutting-edge AI tools to generate leverage in a fast-changing industry and maintain our competitive edge in AI benchmarking

Qualifications

  • 3+ years of relevant professional experience, across industry or research
  • Strong analytical and critical thinking skills
  • Strong Python, with hands-on experience running evaluation harnesses and building datasets
  • Deep familiarity with the LLM evaluation landscape: the major benchmarks and their failure modes, contamination, preference-based methods, and agentic evaluation
  • Strong statistical grounding: you know when a result is signal and when it is noise
  • Genuine, demonstrable interest and knowledge of Frontier AI. We want people who have informed opinions about where AI is heading, not just people who use AI tools

Skills

Required:PythonLLM evaluationStatistical analysisBenchmarkingResearch mindset

Benefits

  • Competitive compensation including equity
Last updated Oct 1, 2026. Interested? Apply on the company site.

About Artificial Analysis

Independent analysis of AI

AI ModelAPI ProvidersAnalysisBenchmarkingIndependenceEvaluationIntelligence IndexLeaderboards
Company profileWebsite

More open roles at Artificial Analysis