← All roles

Member of Technical Staff - Data Quality Engineer (Post-training)

Reflection· Research
Apply on company site
San Francisco, CA, London, New York, NYFull-timePosted 9mo ago

About the role

Data quality engineer focused on upstream data quality for LLM post-training and evaluation, shaping quality standards and scalable QA methods to ensure high-quality data for model training and evaluation.

Responsibilities

  • Own upstream data quality for LLM post-training and evaluation by analyzing expert-developed datasets and operationalizing quality standards for reasoning, alignment, and agentic use cases
  • Partner closely with research and post-training teams to translate requirements into measurable quality signals, and provide actionable feedback to external data vendors
  • Design, validate, and scale automated QA methods, including LLM-as-a-Judge frameworks, to reliably measure data quality across large campaigns
  • Build reusable QA pipelines that reliably deliver high-quality data to post-training teams for model training and evaluation
  • Monitor and report on data quality over time, driving continuous iteration on quality standards, processes, and acceptance criteria

Qualifications

  • Strong engineering fundamentals with experience building data pipelines, QA systems, or evaluation workflows for post-training data and agentic environments
  • Detail-oriented with an analytical mindset, able to identify failure modes, inconsistencies, and subtle issues that affect data quality
  • Solid understanding of how data quality impacts training (SFT and RL) and evaluation, with the ability to translate quality concerns into concrete signals, decisions, and feedback
  • Experience designing and validating automated quality checks, including rule-based systems, statistical methods, or model-assisted approaches such as LLM-as-a-Judge
  • Comfortable working autonomously, owning problems end-to-end, and collaborating effectively with researchers, engineers, and operations partners

Skills

Required:PythonLarge datasets and automated evaluationLLMs understandingCommunication

Benefits

  • Top-tier compensation
  • Stock options
  • Health & wellness: medical, dental, vision, and life, with an annual wellness allowance
  • Meals: Lunch and dinner provided in the office daily
  • Life & family: 22 weeks paid parental leave for all new birthing and non-birthing parents, including adoptive and surrogate journeys
  • Vacation days: Unlimited PTO in the U.S. and 30 days in the U.K
  • Sponsorship support: Visa sponsorship and immigration pathways where applicable
  • Team building: Regular off-sites, happy hours, and team celebrations
Last updated Oct 1, 2026. Interested? Apply on the company site.

About Reflection

Building frontier open intelligence

Building frontier open intelligence.

liquid-ai-recruiting-target
Company profileWebsite

More open roles at Reflection