← All roles

Machine Learning Research Scientist, Evaluations

Scale AI· Research
Apply on company site
Hybrid (San Francisco, CA), Hybrid (Seattle, WA), Hybrid (New York, NY)$165.6k - $207kFull-timePosted 1mo ago

About the role

Role focusing on building benchmarks and diagnosing failure modes in frontier LLMs and agents, with emphasis on post-training techniques (SFT, RLHF, reward modeling) and evaluation across text and multimodal modalities. Collaborates with researchers and engineers to define best practices in evaluation-driven AI development and translates failure analysis into input for next-gen generative AI models.

Responsibilities

  • Analyze model behavior to identify, characterize, and diagnose failure modes in frontier LLMs and Agents across capability gaps, reasoning errors, robustness, and alignment (RCA focus)
  • Design and build benchmarks and evaluation methods measuring LLM capabilities in text and multimodal modalities
  • Apply post-training expertise (SFT, RLHF, reward modeling) to connect observed failures to data/training interventions
  • Publish research findings in top-tier AI conferences

Qualifications

  • Ph.D. or Master's degree in Computer Science, Machine Learning, AI, or related field
  • Deep understanding of deep learning, reinforcement learning, and large-scale model fine-tuning
  • Experience with post-training techniques such as RLHF, preference modeling, or instruction tuning, and with LLM evaluation or benchmark development
  • Excellent written and verbal communication skills
  • Published research in machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals
  • Previous experience in a customer facing role

Skills

Required:Post-training techniques (SFT, RLHF, reward modeling)LLM evaluation and benchmark developmentMultimodal evaluationResearch communication

Benefits

  • comprehensive health, dental and vision coverage
  • retirement benefits
  • learning and development stipend
  • generous PTO
  • potential commuter stipend
Last updated Sep 16, 2026. Interested? Apply on the company site.
Scale AI logo

About Scale AI

Data-centric infrastructure to accelerate the development of AI

Scale accelerates the development of AI within organizations of any size to deliver critical business insights and operational efficiency. Its data-centric infrastructure platform leverages RLHF (Reinforced Learning with Human Feedback) to help organizations build the strongest AI models that supercharge their business, with customers across industries including Meta, Microsoft, U.S. Army, DoD’s Defense Innovation Unit, Open AI, General Motors, Toyota Research Institute, Brex, Instacart and Flexport.

artificial-intelligenceliquid-ai-recruiting-targetmachine-learning
Company profileWebsite

More open roles at Scale AI