← All roles

Principal Security Research Manager

Microsoft· Security Research
Apply on company site
Redmond, WA$142.8k - $274.8kfull-timePosted 2mo ago

About the role

Role leading end-to-end evaluation systems for agentic security and engineering workflows, building the common evaluation platform, methodology, and evidence-based release decisions for MSec agent programs.

Responsibilities

  • Design and build end-to-end evaluation harnesses for agentic security and engineering workflows, including triage, remediation, repo readiness, escalation, tool use, and scan-to-verified-closure paths.
  • Create representative benchmark suites and golden datasets that include normal, edge, adversarial, failure-recovery, regression, and production-derived cases.
  • Implement deterministic validators, automated graders, trace analyzers, result stores, comparison views, and workflow adapters that make evaluations repeatable and actionable.
  • Measure task success, correctness, safety and policy compliance, failure recovery, latency, tool-call behavior, token usage, total cost per successful outcome, and human-review effort.
  • Compare models, prompts, tools, memory strategies, policies, and orchestration patterns under consistent conditions and help teams understand quality-versus-efficiency tradeoffs.
  • Integrate evaluations into engineering workflows, CI/CD, release gates, and decision processes so material agent changes are supported by reproducible evidence before production rollout.
  • Connect offline evaluation results with production feedback, including accepted and rejected outputs, human overrides, incidents, rollbacks, escaped defects, and customer or service-health signals.
  • Detect regressions, benchmark drift, evaluator miscalibration, and cases where eval scores improve while real-world outcomes do not.
  • Partner with agent builders, product teams, security engineers, data scientists, program managers, and leadership to turn evaluation results into release recommendations and autonomy-boundary decisions.
  • Convert learnings into reusable paved paths: playbooks, templates, onboarding guides, dashboards, scorecards, and reference implementations that can scale across MSec.

Qualifications

  • Doctorate in Statistics, Mathematics, Computer Science, Computer Security, or related field AND 3+ years experience in software development lifecycle, large-scale computing, threat analysis or modeling, cybersecurity, vulnerability research, and/or anomaly detection OR Master's Degree in Statistics, Mathematics, Computer Science, Computer Security, or related field AND 4+ years experience in software development lifecycle, large-scale computing, threat analysis or modeling, cybersecurity, vulnerability research, and/or anomaly detection OR Bachelor's Degree in Statistics, Mathematics, Computer Science, Computer Security, or related field AND 6+ years experience in software development lifecycle, large-scale computing, threat analysis or modeling, cybersecurity, vulnerability research, and/or anomaly detection OR equivalent experience.
  • 1+ year(s) people management experience.

Skills

Required:Evaluation system designProduction systems and toolingExperimentation and analysisSecurity engineering and complianceLLMs and AI agents
Last updated Jul 30, 2026. Interested? Apply on the company site.

About Microsoft

Cloud ComputingConsumer ElectronicsSoftwareTechnologyliquid-ai-recruiting-target
Company profileWebsite

More open roles at Microsoft