← All roles

Evaluation Science Lead

Apple· Globalization Quality and Operations
Apply on company site
Cupertino, CA, Austin, TX$175.5k - $311.7kfull-timePosted 2w ago

About the role

Lead the measurement and validation of multilingual Globalization AI solutions across Apple Services (e.g., Apple Music, App Store, Apple TV+, Apple Podcasts) and define evaluation science strategy, frameworks, and best practices for 50+ languages and multiple markets.

Responsibilities

  • Framework & Methodology Design - Define the long-term evaluation science strategy and roadmap for Globalization AI solutions across 50+ languages, including low-resource locales.
  • Design robust statistical methodologies—including power analysis, sampling, confidence intervals, and significance thresholds—to evaluate models with scientific validity.
  • Architect scalable evaluation workflows blending human annotation (calibration, protocols, inter-annotator agreement) with automated systems (autograders, LLM-as-judge, and agent-based first-pass scoring).
  • Expand evaluation criteria beyond core linguistic quality to incorporate behavioral and user engagement signals.
  • Insights & Continuous Improvement - Translate complex evaluation data and loss patterns into clear, actionable recommendations and go/no-go evidence for Engineering and leadership.
  • Build continuous performance monitoring and drift-detection systems to distinguish real regressions from measurement noise.
  • Establish feedback loops with Quality Operations and Engineering to rapidly refine evaluation rubrics as models evolve.
  • Serve as the resident authority on multilingual AI evaluation, communicating methodologies and upleveling best practices across Globalization Apple Services.

Qualifications

  • 5+ years of experience in evaluation science, data science, or ML systems development, with demonstrated experience owning evaluation systems at scale end-to-end
  • Experience applying statistical methodology — sampling, significance testing, and confidence intervals
  • Practical understanding of measurement validity principles
  • Hands-on experience measuring annotator agreement, diagnosing divergence, and improving annotation protocols
  • Proficiency with statistical tools and languages (R, Python, SQL) for analysis and reproducibility
  • Experience designing evaluation frameworks adopted and scaled by operational teams
  • Strong written and verbal communication skills for technical and non-technical audiences
  • Direct experience handling sensitive and confidential information with integrity and discretion
  • Ability to be onsite; this role is an in-person, onsite position
  • Availability to work occasional evenings and weekends, as business needs require
  • Up to 10% + travel; both domestic and international
  • Master’s, PhD, or comparable experience in Statistics, Computational Linguistics, Computer Science, Psychometrics, Data Science, or related quantitative field
  • Experience evaluating generative AI systems—including hallucination detection, safety and cultural alignment, autograders / LLM-as-judge systems, benchmark design, and synthetic data evaluation
  • Experience with A/B testing, causal inference, or experimental design
  • Experience in applied linguistics — translating cultural and linguistic nuances into quantitative evaluation framework

Skills

Required:Statistical analysisExperimental design / A/B testingAnnotation quality & protocol designCommunication

Benefits

  • Comprehensive medical and dental coverage
  • retirement benefits
  • discounted products and free services
  • reimbursement for certain educational expenses including tuition
  • discretionary bonuses or commission payments as well as relocation (where eligible)
Last updated Oct 9, 2026. Interested? Apply on the company site.

About Apple

Discover the innovative world of Apple and shop everything iPhone, iPad, Apple Watch, Mac, and Apple TV, plus explore accessories, entertainment, and expert device support.

Discover the innovative world of Apple and shop everything iPhone, iPad, Apple Watch, Mac, and Apple TV, plus explore accessories, entertainment, and expert device support.

AirPods Max 2Apple ArcadeApple CardApple Fitness+Apple MusicApple TVApple TV+Apple Watch Series 11
Company profileWebsite

More open roles at Apple