← All roles

Research Engineer, Evals - Member of Technical Staff

Callosum· General Capabilities
Apply on company site
LondonFull-timePosted 2mo ago

About the role

Role focused on designing and implementing agentic evaluation benchmarks and observability for heterogeneous AI systems, to provide principled, evidence-based measurements that localize failures and inform system design.

Responsibilities

  • Design benchmarks and evaluation suites for agentic behaviour – multi-turn, long-horizon, tool-using, operating in non-stationary environments – including properties hard to score such as reliability and recovery from error
  • Build the methodology as well as the harness: statistical power, variance across runs, contamination and construct validity
  • Be the independent voice on the quality of our own results. Red-team evaluations, design sanity checks that catch flattering numbers before they reach a decision
  • Turn raw traces into structured evidence. Failure taxonomies, behavioural signatures, and attribution of an outcome to the decision that caused it
  • Move from measurement to prediction: infer what a system can do from partial evidence, and estimate performance on a task before running it
  • Build evaluation and observability infrastructure as durable instruments the whole company works from

Qualifications

  • Evidence that you can run research of your own: you take an open question, design the experiments that settle it, and produce results other people can build on
  • Deep hands-on experience with LLMs in agentic settings: multi-step tasks, tool use, long horizons, and the specific ways all of it breaks
  • Statistical discipline. Hypotheses stated in advance, uncertainty quantified, and running comparisons that isolate one variable
  • Strong engineering. You write the code that runs your experiments and you are comfortable working inside a large shared codebase
  • Strong communication skills. You are able to turn research findings into clear, prioritised guidance for the teams who will act on them, and to write them up for a wider audience

Skills

Required:Experiment designAgentic LLMsStatistics and uncertainty quantificationEngineeringCommunication

Benefits

  • Competitive Salary
  • Equity & Ownership
  • Private healthcare
  • Visa sponsorship and relocation benefits
  • In-person London office with supportive setup
Last updated Oct 14, 2026. Interested? Apply on the company site.

About Callosum

The Intelligent Systems Company.

Callosum - Co-evolving chips & intelligence

AISystems companychipscomputedevelopmentheadline-ai-europe-100-2026heterogeneousinfrastructure
Company profileWebsite

More open roles at Callosum