← All roles

Principal Applied Scientist for Copilot Evals

Microsoft· Applied Sciences
Apply on company site
Redmond, WA$142.8k - $274.8kfull-timePosted 1mo ago

About the role

Principal Applied Scientist to lead customer-grounded evaluation and learning signals for Copilot, translating enterprise customer needs into reusable evals, RLEs, reward signals, and post-training priorities; set scientific direction for evaluation quality and collaborate across applied researchers and ML engineers.

Responsibilities

  • Set the scientific strategy for customer-grounded quality across priority Copilot intents, defining what good means and the tradeoffs across various quality and safety attributes
  • Translate user research, enterprise customer feedback, DSAT, and production incidents into evaluation and post-training priorities, then lead cross-team creation of reusable evaluation, regression, RLE, and post-training assets for the highest-value workflows and failure patterns.
  • Develop methods to assess evaluation-set representativeness, coverage, freshness, discrimination, grader reliability, and alignment with production outcomes, identifying material gaps, drift, and emerging loss patterns.
  • Design behavior and task evaluations, rubrics, graders, and calibration methods that translate qualitative customer expectations into measurable release-over-release quality.
  • Establish methods to attribute quality losses across grounding, retrieval, tools, orchestration, model reasoning, response generation, and evaluation, linking offline movement with online signals such as DSAT, task completion, retries, abandonment, and escalation.
  • Set a high bar for scientific rigor, reproducibility, documentation, and interpretation of results while mentoring scientists and engineers and influencing evaluation and post-training strategy across organizational boundaries.

Qualifications

  • Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 6+ years related experience (e.g., statistics, predictive analytics, research) OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 4+ years related experience (e.g., statistics, predictive analytics, research) OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 3+ years related experience (e.g., statistics, predictive analytics, research) OR equivalent experience.

Skills

Required:Statistical inference, sampling, measurement validity, bias, uncertainty, experimental designCommunication across organizations
Nice to have:Evaluation design and metrics
Last updated Aug 27, 2026. Interested? Apply on the company site.

About Microsoft

Cloud ComputingConsumer ElectronicsSoftwareTechnologyliquid-ai-recruiting-target
Company profileWebsite

More open roles at Microsoft