Senior Product Operations Manager, Evaluation Quality
Hybrid (San Francisco)$155.4k - $233.2kFull-timePosted 2mo ago
About the role
Senior operator responsible for defining and maintaining the quality bar for Harvey’s human evaluations, driving decision-grade outputs, and ensuring scalable, reproducible evaluation quality across jurisdictions and geographies.
Responsibilities
- Own the quality bar for Harvey’s human evaluations: define what “good” looks like for eval methodology and data analysis, and produce decision-grade outputs for EPD
- Author and maintain the evaluation guidelines, instructions, and databases that contract attorneys work from
- Standardize and streamline rubric and evaluation design into repeatable templates and one documented methodology, partnering with Applied Legal Research (ALR)
- Own contract-attorney quality: onboarding, calibration training, inter-rater reliability, and the feedback loop that keeps judgment consistent across attorneys and over time
- Conduct quantitative and qualitative statistical analyses, diagnosing error states, investigating root causes, and turning raw eval results into a structured, prioritized signal Product and ALR can act on
- Run QA on vendor and contract-attorney deliverables against a defined bar before results inform a launch decision
- Ensure the quality bar holds across jurisdictions and non-English geographies as coverage expands
- Establish one standard, documented way to analyze eval results, and build lightweight operational dashboards to track rater capability and eval-program health
- Support ALR in a review step that certifies an evaluation is sound before it scales to contract attorneys
- Partner with ALR and Analytics to determine where human eval aligns with online signal and where it can provide expanded insights
Qualifications
- 6+ years in product operations, research operations, evaluation/QA operations, or quality program management
- A track record of owning quality inputs for complex, expert-driven or human-in-the-loop work
- Experience onboarding, training, and calibrating a distributed pool of expert raters, annotators, or reviewers, and running the feedback loop that improves their quality over time
- Grounding in measurement concepts (calibration, inter-rater reliability, sampling, rubric design) to independently set up and own the quality of the evaluation loop
- Experience with running quantitative and qualitative data analyses, interpreting and running statistical tests on evaluation data, and communicating conclusions to stakeholders
- A record of scaling and streamlining evaluation quality processes under shipping pressure, prioritizing documentation and reproducibility
- Ability to work deeply with domain experts and translate nuanced judgment into repeatable standards
- Strong cross-functional coordination across Product, Engineering, Research, ALR, and data providers/vendors
- Clear communicator with credibility and trust-building, bias to action and ownership
Skills
Required:Quality governance for evaluation dataGuideline and rubric designStatistical analysis (quantitative)Qualitative analysisOnboarding and calibration of raters
Benefits
- Comprehensive health, dental and vision coverage
- 401k match up to 4%
- Flexible PTO
Last updated Aug 12, 2026. Interested? Apply on the company site.
About Harvey
Professional Class AI
Harvey is a platform built to meet the standards of the world’s leading professional service firms.
AILegalLegal AILegal AnalysisLegal Document AnalysisLegal ResearchLegal Techsetter-30
More open roles at Harvey
- Senior Deal Desk Analyst, APAC4d agoSydney · Sales
- Deal Operations Analyst4d agoHybrid (Chicago) · Sales
- Deal Operations Analyst4d agoHybrid (Dallas) · Sales
- Account Executive, Enterprise4d agoHybrid (Toronto) · Sales
- Head of Mid Market Sales, North America4d agoHybrid (Chicago) · Sales