← All roles

AI Evaluation Infrastructure Engineer

Block· 10418 Engineering - Data & Data Science
Apply on company site
Hybrid (Bay Area)$263.6k - $395.4kfull-timePosted 2w ago

About the role

Engineer to build infrastructure and tooling enabling high-quality AI evaluation at Block's scale, integrating offline and online evals with feedback loops to improve AI products.

Responsibilities

  • Build an execution engine that can score candidate versions against task sets in minutes, not hours.
  • Create task set tooling that samples from production logs and validates tasks before they are admitted into an evaluation set.
  • Build grader infrastructure across ground truth checks, rubrics, and LLM-as-judge approaches.
  • Develop tooling that helps human reviewers calibrate judges, measure judge-to-human agreement, and monitor drift over time.
  • Build leaderboards and reporting systems that include sample size, confidence intervals, and run-to-run variance, so teams can distinguish real improvements from noise.
  • Support in-product side-by-side serving, feedback capture, and implicit signal extraction from real conversations.
  • Build the loop that compares offline scores with online outcomes, identifies eval sets that have stopped predicting reality, and helps teams improve them.
  • Partner with product, engineering, data, and ML teams to make evaluation workflows fast enough and trustworthy enough to become part of everyday development.

Qualifications

  • Experience building production platforms or infrastructure, including distributed batch execution, data pipelines, or systems that process production logs.
  • Strong statistical literacy, including comfort with confidence intervals, variance, power, and multiple comparisons.
  • The judgment to identify when a result is meaningful and when it is noise.
  • Experience evaluating LLM or ML systems, or deep systems engineering experience with a strong interest in AI evaluation.
  • Product instinct for internal tools. You understand that leaderboards, annotation tools, and workflows only matter if teams actually use them.
  • A bias toward building reliable, observable systems that other engineers can trust.
  • Strong collaboration skills and the ability to work across ambiguous product, data, and engineering problems.

Skills

Required:Production platforms / infrastructureStatistical literacyLLM/ML evaluation

Benefits

  • Remote work
  • medical insurance
  • flexible time off
  • retirement savings plans
  • modern family planning
  • Check out our other benefits at Block.
Last updated Oct 7, 2026. Interested? Apply on the company site.
Block logo

About Block

Block builds technology to increase access to the global economy.

Made up of Square, Cash App, Afterpay, TIDAL, Bitkey, and Proto, Block, Inc. builds technology to increase access to the global economy.

technologyfinancial servicesdigital paymentsmusic streaming
Company profileWebsite

More open roles at Block