Member of Technical Staff, Evals
San Francisco, CA, Remote (USA), New York, NY, Mountain View, CA$200k - $350kFull-timePosted 3w ago
About the role
Role focused on defining how frontier AI systems are measured, understood, and improved. High-ownership position for researchers who build, partnering with AI researchers, domain experts, and customers to develop benchmarks, reward/verifier systems, and data-quality techniques. Early team members influence technical direction, open-source software, benchmarks, and research products.
Responsibilities
- Design and build evaluation frameworks, benchmarks, and methodologies for frontier LLMs, AI agents, multimodal models, and reinforcement-learning environments.
- Develop reward models, programmatic verifiers, graders, and other feedback systems that make model behavior measurable and improvable.
- Research what makes evaluations representative, difficult, reliable, and resistant to shortcutting or reward hacking.
- Build systems for high-quality human data, including expert task design, annotation methodologies, data-quality signals, and data-attribution techniques.
- Run fast, rigorous iteration loops: prototype, evaluate, interpret results, diagnose failure modes, and turn learnings into the next benchmark or system.
- Publicly contribute to the field through benchmarks, open-source tools, research, and technical writing.
Qualifications
- PhD in ML/AI, computer science, data science, or related fields (or equivalent research experience in industry).
- Publications at top AI/ML venues like NeurIPS, ICML, ICLR, COLM.
- Builders who enjoy tinkering with agents and shipping high-quality software, benchmarks, or datasets (e.g. a strong GitHub profile / OSS contributions, or product portfolio).
- Strong Python skills, experience building scalable software, working with agents.
- Strong knowledge of frontier AI: benchmarks, eval techniques, agent harnesses, post-training recipes, data shapes.
- Comfort operating in an ambiguous, fast-moving environment with substantial ownership.
Skills
Required:PythonResearch & Publications
Benefits
- Ownership: Equity in a fast-growing company
- Financial Wellness: 401(k) match, competitive compensation, financial coaching
- Family Support: Paid parental leave, fertility benefits, parental coaching
- Wellbeing: Medical, dental, and vision, mental health support, $500 wellness stipend
- Growth: $2,000 learning stipend, ongoing development
- Office: Commuting support, free lunch, and gym in our SF office
- Time Off: Flexible PTO, 15 holidays + 2 flex days
- Connection: Team outings & referral bonuses
Last updated Oct 1, 2026. Interested? Apply on the company site.
About Handshake
Careers start here
The better career platform for Gen Z changing how, where, and why the next generation of talent builds their career.
educationcareer platformtalent acquisitionstudent engagement
More open roles at Handshake
- Privacy Associate5d agoSan Francisco, CA · General & Administrative
- Seattle Site Lead5d agoSeattle, WA · HAI Delivery Ops
- Senior Recruiter5d agoSan Francisco, CA · General & Administrative
- Director, Product Marketing5d agoHybrid (San Francisco, CA) · Marketing
- AI Product Counsel5d agoSan Francisco, CA · General & Administrative