← All roles

Applied AI Researcher, Benchmarking

Distyl AI· Research
Apply on company site
Hybrid (San Francisco), Hybrid (New York)$150k - $250kFull-timePosted 11mo ago

About the role

Research-focused role developing evaluation frameworks and benchmarks for intelligent systems to measure progress, reliability, and operational impact in enterprise AI applications.

Responsibilities

  • The Benchmarking team defines how progress is measured. Researchers design evaluation frameworks that capture reasoning depth, interaction quality, reliability, and operational impact. They construct benchmarks that reflect real-world complexity. Their systems become the standard by which new architectures, techniques, and releases are judged.
  • Researchers in Benchmarking explore new paradigms for evaluating intelligent systems: adversarial robustness testing, longitudinal performance tracking, and human-in-the-loop assessment. They investigate how metrics shape model behavior and establish rigorous methodologies for quantifying emergent capability. Their insights drive both Distyl’s internal research priorities and industry-wide standards.

Qualifications

  • Experience Designing and Running Evaluations: You’ve built or maintained benchmarks, test suites, or experimental frameworks to measure model or system performance
  • Statistical and Analytical Rigor: You design fair, reproducible experiments and can extract signal from noisy empirical results
  • Experience Building with Models, Not Just Building Models: Expertise in compound AI systems, agentic collaboration, and techniques (ensembling, ReAct, graph-of-thoughts, etc.)
  • Proven Track Record of Research Results: Published in top journals or notable work shared publicly
  • Uses AI Every Day: Comfortable using tools like ChatGPT, Cursor, and Perplexity to accelerate workflow
  • Strong Programming and Data Analysis Skills: Able to build prototypes and run experiments to prove effectiveness to a Fortune 500 Head of AI
  • Biases Towards Showing vs Telling: Demonstrate impact and capabilities to customers today

Skills

Required:Experience Designing and Running EvaluationsStatistical and Analytical RigorExperience Building with Models, Not Just Building ModelsProven Track Record of Research ResultsUses AI Every DayStrong Programming and Data Analysis Skills

Benefits

  • 100% coverage of medical, dental, and vision insurance for employee and dependents
  • Flexible time off
  • Retirement and financial planning benefits (pre-tax HSA, FSA, commuter accounts, 401(k), financial coaching)
  • Wellness benefits (physical fitness, mental well-being, fertility and family-building benefits)
  • Complimentary in-office lunches and snacks
  • Access to state-of-the-art AI models and tools
  • Ownership of high-impact projects across top enterprises
  • Mission-driven, fast-moving culture
Last updated Aug 12, 2026. Interested? Apply on the company site.

About Distyl AI

Rearchitecting Entire Industries For Frontier Technologies

Building AI solutions that power F500 big bets and create a competitive edge.

AIIndustry transformationEnterprise AIInnovation
Company profileWebsite

More open roles at Distyl AI