← All roles

Machine Learning Evaluation Engineer

Apple· Hardware
Apply on company site
Sunnyvale, CA$150.4k - $277.6kfull-timePosted 2w ago

About the role

Role focused on designing and deploying evaluation strategies for multi-modal foundation models and CV/ML algorithms, including benchmarking, data preparation, failure analysis, and automation, to ensure high-quality models for Apple products.

Responsibilities

  • Algorithm Evaluation & Benchmarking: Design and build comprehensive evaluation pipelines that scale across large datasets. You will be responsible for both holistic end-to-end system evaluation and granular component-level testing to rigorously measure model capabilities on complex text, image, and video understanding tasks.
  • Subjective Evaluation: Solve the challenge of developing and calibrating subjective evaluations for generative models by leveraging LLM-as-Judge and human grading techniques.
  • Deep Failure Analysis: Identify trends in large datasets and dive deep into specific failure cases to find their root causes in models, prompts, or upstream software and algorithm components. Develop failure taxonomies to track over time.
  • Workflow Automation: Design and develop cloud-based, LLM-powered workflows and dashboards to streamline failure analysis, reporting, and iterative experimentation.
  • Evaluation Set Curation: Refine our strategy for curating high-quality datasets and ground truth, whether through auto-labeling with VLMs, leveraging manual annotation teams, or generating synthetic data.
  • Data Analysis: assess quality, diversity, representativeness, and coverage gaps across large, complex datasets to ensure evaluation sets reflect real-world usage.
  • Cross-Functional Collaboration: Partner closely with the core model training teams to provide actionable, data-driven insights and metrics to guide the next iteration of model training and fine-tuning.

Qualifications

  • BS and a minimum of 3 years relevant industry experience
  • 3+ years of applied experience in Machine Learning, Computer Vision, or AI System Evaluation.
  • Deep understanding of core Machine Learning principles, including probability, statistics, data distributions, and model bias/variance. Ability to apply statistical rigor to ensure evaluation metrics are meaningful and reliable.
  • Experience analyzing data quality, diversity, representativeness, and coverage gaps in large datasets, and translating findings into data collection and curation strategies.
  • Applied experience employing and evaluating generative models, including knowledge of prompt tuning, subjective evaluation of open-ended outputs, and human-in-the-loop methodologies.
  • Proven track record of defining robust metrics/KPIs and designing rigorous evaluation frameworks for foundation models. Deep experience with custom benchmark creation, automated regression testing, and LLM/VLM-as-a-Judge methodologies.
  • Strong intuition for probing ML models to discover edge cases, hallucinations, and performance bottlenecks in constrained environments. Ability to translate findings into actionable recommendations for model improvement.
  • Strong proficiency in Python and building well-designed automated pipelines, including model endpoint integration, multi-step evaluation orchestration, and robust logging for analysis and traceability.
  • Experience using AI-powered development and analysis tools to accelerate data exploration, coding, and workflow automation, while maintaining technical rigor and reproducibility.
  • Theoretical and practical understanding of Computer Vision (CV) and Vision-Language Models (VLMs) in order to effectively anticipate failure modes and accurately benchmark their performance.
  • Excellent written and verbal communication skills, with the ability to describe complex topics to cross-functional teams with varying levels of technical expertise.

Skills

Required:Algorithm Evaluation & BenchmarkingSubjective EvaluationDeep Failure AnalysisWorkflow AutomationEvaluation Set CurationData Analysis

Benefits

  • Comprehensive medical and dental coverage
  • Retirement benefits
  • Discounted products and free services
  • Education-related reimbursement (tuition)
  • Discretionary bonuses or commission potential
  • Relocation assistance
  • Apple stock programs (RSUs, Employee Stock Purchase Plan) available to eligible employees
Last updated Oct 9, 2026. Interested? Apply on the company site.

About Apple

Discover the innovative world of Apple and shop everything iPhone, iPad, Apple Watch, Mac, and Apple TV, plus explore accessories, entertainment, and expert device support.

Discover the innovative world of Apple and shop everything iPhone, iPad, Apple Watch, Mac, and Apple TV, plus explore accessories, entertainment, and expert device support.

AirPods Max 2Apple ArcadeApple CardApple Fitness+Apple MusicApple TVApple TV+Apple Watch Series 11
Company profileWebsite

More open roles at Apple