← All roles

Benchmarking Project Lead, Siri Evaluation

Apple· Machine Learning and AI
Apply on company site
Cambridgefull-timePosted 1mo ago

About the role

Senior manager role within the Siri Evaluation team focused on driving evaluation efforts for Siri AI. Lead data collection, annotation, budgeting, and tooling initiatives to benchmark and improve agentic evaluation across features, locales, and devices, with humans-in-the-loop methodologies and cross-functional collaboration.

Responsibilities

  • Design and execute efficient data collection processes using humans in the loop
  • Lead annotation efforts for various languages and devices
  • Plan and manage the budget of annotator resourcing with Finance, Annotation Ops, and International team partners
  • Track and improve the quality of the human judgements through revision of annotator training materials, clear annotation questions, annotator training, and efficient reviewing mechanisms
  • Collaborate with other engineering teams to design and build a tooling ecosystem for managing and browsing rich datasets

Qualifications

  • Agentic Coding proficiency to achieve data-science, data collection and visualisation tasks
  • Good understanding of metrics, crowd science, data collection, annotation analysis, statistics
  • Ability to work independently and cross-functionally to integrate in partner team reporting systems and pipelines
  • Excellent communication skills and the ability to thrive in a highly collaborative work environment
  • Attunement to computational linguistics, language quality, human in the loop evaluation
  • Good engineering practices to create sustainable and easy to use data management pipelines
  • Python experience and other tools for data collection and visualisation

Skills

Required:Data collectionAnnotation managementBudget planningData quality controlCross-functional collaborationPython

Benefits

  • Learn about accessibility and inclusive hiring at Apple
  • Disability Confident employer accommodations
  • Competitive benefits typical of Apple roles (not explicitly listed in description)
Last updated Oct 16, 2026. Interested? Apply on the company site.

About Apple

Discover the innovative world of Apple and shop everything iPhone, iPad, Apple Watch, Mac, and Apple TV, plus explore accessories, entertainment, and expert device support.

Discover the innovative world of Apple and shop everything iPhone, iPad, Apple Watch, Mac, and Apple TV, plus explore accessories, entertainment, and expert device support.

AirPods Max 2Apple ArcadeApple CardApple Fitness+Apple MusicApple TVApple TV+Apple Watch Series 11
Company profileWebsite

More open roles at Apple