Technical Program Manager - Frontier AI Data Labeling
Hippocratic AI· R&D
Menlo Park, CAFull-timePosted 4d ago
About the role
Technical Program Manager to run the human-in-the-loop data labeling engine behind Hippocratic AI's frontier healthcare AI, spanning audio and conversation annotation, clinician-labeled training data, preference and RLHF data, and evaluations that decide what ships.
Responsibilities
- Run the data pipelines behind our models. Own end-to-end delivery of labeling and annotation programs, including audio and voice-conversation labeling, supervised labeling, and preference and RLHF data. Manage scoping, task design, execution, QA, and delivery against live model-development cycles.
- Turn evals into products. Work with ML and Applied Science to turn evaluation and training-signal needs into concrete labeling tasks, rubrics, and success criteria. Evals become the benchmarks that gate releases.
- Build and scale an expert rater workforce. Source, screen, and onboard clinicians and specialized raters through marketplaces, vendors, and in-house teams. Design qualification tasks, set rates, and scale pools as programs change.
- Write the guidelines that shape model behavior. Author and iterate labeling guidelines and annotation rubrics, especially for subjective, high-stakes clinical judgment.
- Engineer data quality at scale. Design QC systems using gold sets, calibration sessions, spot checks, and inter-rater agreement tracking. Catch rater drift and annotation bottlenecks before they reach training.
- Drive velocity and accountability. Track throughput, cost, and quality. Manage vendor relationships (SOWs, performance reviews), choose labeling and QA tooling, and align Engineering, Clinical, and Infrastructure on dependencies and timelines.
Qualifications
- 5+ years in technical program management or data operations in an AI/ML or data-intensive environment (3+ years considered for exceptional candidates)
- 2+ years hands-on with data labeling, annotation, or model evaluation, including writing guidelines and managing quality at scale
- Experience sourcing and managing contractor or expert annotator workforces through marketplaces or labeling vendors
- Working knowledge of how annotation quality affects model training, including preference data, pairwise comparisons, and rater calibration (RLHF)
- A strong grasp of quality metrics (IAA, precision/recall on gold sets, throughput vs. accuracy tradeoffs)
- Track record of owning critical data or eval projects under time pressure, collaborating with ML engineers and applied scientists, and delivering results without direct authority
- Comfort with ambiguity in a fast-moving, healthcare-adjacent AI environment
Skills
Required:Data labeling and annotation program managementEvaluation-to-product workflowRater workforce development
Benefits
- (Not explicitly listed in the description)
Last updated Oct 21, 2026. Interested? Apply on the company site.
About Hippocratic AI
Safest Healthcare Generative AI Agents
Safest Healthcare Generative AI Agents Our AI agents do not diagnose or prescribe. Learn more. Listen to these 4 real de-identified calls from the over 115 million clinical interactions.
HealthcareAIGenerative AIClinical InteractionsPatient Care
More open roles at Hippocratic AI
- Governance Risk Compliance4d agoMenlo Park, CA · G&A
- Lead Product Data Analyst4d agoMenlo Park, CA · R&D
- Deployment Strategist (Payors)1w agoRemote · C&S
- Accounting Manager1w agoGurugram, India · G&A
- RN Clinical Data Annotation Specialist1w agoRemote · Clinical