Research Engineer, Evals
About the role
Role focused on building benchmarks, datasets, tooling, and evaluation loops to measure and improve model quality for AI agents in risk investigations, fraud review, and identity reviews at Variance (SF-based). Centered in research, product, and engineering to drive domain-specific evaluations and robust system quality.
Responsibilities
- Build proprietary benchmarks and datasets to evaluate models and model systems on fraud, identity, and risk workflows
- Design and run offline and online evals that measure model performance on real customer tasks, not just abstract benchmarks
- Define quality metrics for judgment systems, including precision, calibration, consistency, abstention, and failure handling
- Study where models and agents break, and turn those failures into better evals, better datasets, and better training loops
- Build reusable evaluation tools and quality building blocks that can be used across different product surfaces and workflows
- Partner closely with research, engineering, product, and design to improve system quality through rigorous experimentation
- Help create a strong culture of scientific experimentation, clear measurement, and continuous iteration
- Push the boundary of how AI systems are evaluated in regulated, adversarial, and high-consequence environments
Qualifications
- Experience training, evaluating, or improving modern ML systems
- Strong programming skills and comfort working in research-heavy codebases
- Experience building benchmarks, datasets, evaluation pipelines, or quality systems
- Familiarity with LLMs, agent systems, retrieval, post-training, or adjacent areas
- Ability to design clean experiments and draw reliable conclusions from noisy results
- Strong engineering judgment and a bias toward building
- Interest in fraud, risk, trust and safety, compliance, or other regulated and adversarial domains
Skills
Benefits
- Competitive salary and meaningful equity
- Platinum-level medical, dental, and vision insurance
- Unlimited PTO, sick leave, and parental leave
- Up to $100 per month in reimbursement for personal health and wellness expenses
- 401(k) plan
About Variance
World-Class Tooling for Trust & Safety teams
Build safer platforms with Intrinsic. Get access to the best safety technologies from one unified API. Enable your Trust & Safety team to tell cohesive stories from policy to product, to enforcement. Michael and Karine first met two years ago at Apple’s Fraud Engineering, Algorithmic Risk team, which evolved to become Apple’s Trust & Safety function. We are both engineers and helped solve online abuse problems, including spam, botting, account security, and developer fraud for Apple’s customer base of over a billion users.
More open roles at Variance
- Enterprise Account Executive4mo agoSan Francisco, CA · Operations
- Product Marketing Lead4mo agoHybrid (San Francisco)
- Senior Software Engineer4mo agoHybrid (San Francisco) · Engineering
- Research Engineer, Judgment Systems4mo agoSan Francisco · Engineering
- Enterprise Account Executive4mo agoHybrid (New York, NY) · Operations