Senior Applied Scientist, Frontier AI Assets - Assessments (FAI)
Amazon· Applied Science
Hybrid (Bellevue, WA)$167.1k - $226.1kfull-timePosted 2w ago
About the role
Role within Amazon's Frontier AI (FAI) organization focusing on assessment of data quality for frontier models and benchmarks. Develop automated quality assessment systems (LLM-as-a-Judge) and self-improving agents, calibrate against expert judgment, and set technical direction for assessment science.
Responsibilities
- Design the assessment methodology for frontier AI assets
- Build the automated assessment system, from LLM-as-a-Judge baselines to agents that plan their own audits and learn from their own errors
- Oversee the expert audit program and raise the proficiency of the auditors who run it
- Design the measurement methods that make quality findings defensible, including sampling strategy and error analysis
- Communicate findings to the teams that create the data and to the teams that train models and build agents using it
- Set technical direction for assessment science, mentor junior scientists, and present results to senior leadership
- Publish research on assessment methodology at top-tier venues and contribute to patents
- Train AI agents to assess the quality of datasets and evaluation benchmarks, and document where they fall short
- Diagnose their failures and improve the models, rubrics, and calibration behind them
- Review the quality reports that auditors and agents produce, and decide whether the conclusions hold
- Coach expert auditors and junior scientists, and move more of the manual audit work into automation
- Lead research on self-improving assessment agents, from open problem to publication
- FAI Assessment is part of Frontier AI Assets in AGI, focusing on data quality for datasets and benchmarks behind frontier models and agents.
Qualifications
- 5+ years of building machine learning models for business application experience
- Experience programming in Java, C++, Python or related language
- Experience with neural deep learning methods and machine learning
- PhD in Computer Science, Engineering, Mathematics, Statistics, Economics, or a related field
- Experience with Machine Learning and Large Language Model fundamentals, including architecture, training/inference lifecycles, and optimization of model execution, or experience building complex software systems that have been successfully delivered to customers
- Experience in machine learning, data mining, information retrieval, statistics or natural language processing, or experience designing or architecting (design patterns, reliability and scaling) of new and existing systems
Skills
Required:Machine learning model buildingProgrammingDeep learning
Nice to have:Research/academic publication
Benefits
- health insurance (medical, dental, vision, prescription)
- Basic Life & AD&D insurance and option for Supplemental life plans
- EAP, Mental Health Support, Medical Advice Line
- Flexible Spending Accounts
- Adoption and Surrogacy Reimbursement coverage
- 401(k) matching
- paid time off
- parential leave
- sign-on payments and RSUs as part of compensation
Last updated Oct 25, 2026. Interested? Apply on the company site.
More open roles at Amazon
- IT Support Associate II, Ops Tech Solutions (OTS)yesterdayPlainfield, Indiana · Operations IT Support Engineering
- Head of Energy and Water, AWS LegalyesterdayHybrid (Seattle, WA) · Legal
- Software Development Engineer, Creative Asset Library, CreativeXyesterdaySeattle · Software Development
- Software Development Engineer, Creative Asset Library, CreativeXyesterdayHybrid (Seattle, WA) · Software Development
- IT Support Associate II, OTSyesterdayIrving, TX · Operations IT Support Engineering