← All roles

Senior Software Development Engineer in Test — LLM Evaluation & Automation, T3E

Apple· Software and Services
Apply on company site
Hybrid (San Diego, CA), Hybrid (Cupertino, CA)$175k - $308.5kfull-timePosted 2mo ago

About the role

Senior SDET to design and implement automated model evaluation (LLM-as-a-judge) within Apple Intelligence Platform Experience Validation. Focus on building evaluation harnesses, integrating into CI/automation pipelines, and ensuring scalable, actionable quality signals for generative AI features.

Responsibilities

  • Design, build, and maintain LLM-as-a-judge evaluation harnesses and integrate them into existing and new CI/automation pipelines.
  • Author and curate eval sets and rubrics; partner with modeling teams whose own eval sets can run hundreds of examples judged by a separate model.
  • Run eval jobs at scale, triage results, and distinguish real model regressions from rubric problems or infrastructure noise so the signal stays actionable.
  • Plan and build test coverage and tooling from low level component tests through to end-to-end tests that exercise models and frameworks powering Generative AI features
  • Package eval tooling for reuse — reusable libraries and jobs that other engineers on the team and partner teams can adopt.
  • Define quality gates and reporting so model regressions are caught and communicated before human eval or population rollout.
  • Collaborate with data scientists and modeling engineers on approaches to spot regressions across large output sets or between model updates.

Qualifications

  • BS in Computer Science, Mathematics, or a related field (or equivalent practical experience)
  • Three years of relevant industry experience in test automation, software development, or related areas.
  • Strong practical knowledge of Python, including data-pipeline fluency (JSON/YAML, REST APIs).
  • Hands-on experience with LLM-as-a-judge evaluation and rubric design, or a strong demonstrated ability to ramp into it quickly.
  • Strong software engineering fundamentals — able to define atomic, composable components and build maintainable pipelines and tooling, not just scripts.
  • Strong debugging and triage skills; able to separate genuine regressions from infrastructure or rubric noise.
  • Strong knowledge of the software development lifecycle, testing methodologies, and QA processes.
  • Excellent written and verbal communication; able to document clearly and describe quality signal to modeling and leadership audiences.
  • Ability to lead work across varying priorities and partner multi-functionally with modeling, framework, and infrastructure teams.
  • Experience building on-device tooling and device/model eval infrastructure.
  • Experience integrating with CI/CD and job orchestration systems, and comfort deploying tooling as reusable libraries.
  • Familiarity with generative model behavior — image generation, NLP, or LLM output evaluation.
  • Experience curating and reasoning about large datasets; comfort manually inspecting data (Jupyter or similar) to build intuition and drive next steps.
  • Awareness of dataset bias and fairness considerations in evaluation.
  • Experience with database/query tooling (e.g., SQL) and dashboards/visualization for reporting quality trends.
  • Experience with Xcode is a bonus.

Skills

Required:PythonLLM-as-a-judge evaluationSoftware engineering fundamentalsCI/CD and job orchestrationData analysis and debugging

Benefits

  • Comprehensive medical and dental coverage
  • retirement benefits
  • range of discounted products and free services
  • reimbursement for certain educational expenses including tuition
  • possible discretionary bonuses or commission payments
  • relocation assistance
  • Apple Employee Stock Purchase Plan
  • stock programs (discretionary RSUs)
Last updated Aug 7, 2026. Interested? Apply on the company site.

About Apple

Discover the innovative world of Apple and shop everything iPhone, iPad, Apple Watch, Mac, and Apple TV, plus explore accessories, entertainment, and expert device support.

Discover the innovative world of Apple and shop everything iPhone, iPad, Apple Watch, Mac, and Apple TV, plus explore accessories, entertainment, and expert device support.

AirPods Max 2Apple ArcadeApple CardApple Fitness+Apple MusicApple TVApple TV+Apple Watch Series 11
Company profileWebsite

More open roles at Apple