← All roles

AI Engineer, Evals & Agent Quality

Town· EPD
Apply on company site
San Francisco$250k - $300kFull-timePosted 1mo ago

About the role

Town is building an AI assistant that deeply personalizes and operates across tools. The role owns the evals, labeling, model routing, and online measurement to turn assistant performance into measurable numbers and to guide improvements across multi-step agent trajectories.

Responsibilities

  • Build a generalized eval system that measures assistant quality across every surface it touches — and, crucially, across multi-step agent trajectories.
  • Stand up golden datasets and the labeling loop that keeps them up to date and constantly checking to validate improvements and avoid regressions.
  • Build model routing and online evaluation tooling to help us learn and route to the best models.
  • Make every prompt and system change measurable, so the team can move fast without breaking what works.
  • Partner with engineers across the product to instrument quality and close the loop from signal to fix.

Qualifications

  • Have built or owned LLM eval systems, or offline/online quality measurement at scale.
  • Think rigorously about measurement. Maybe that came from an MLE or applied-ML background, maybe not, the instinct for how to measure 'better' matters more than the exact pedigree.
  • Know the eval landscape hands-on, off-the-shelf tooling and eval frameworks, and have opinions on what to reach for when.
  • Are comfortable reasoning about model routing and the tradeoffs between models.
  • Ship the fixes, not just the dashboards and metrics.
  • Are a senior or staff engineer comfortable in greenfield, where the system doesn't exist yet.

Skills

Required:LLM evaluation systemsMeasurement rigorModel routing
Last updated Sep 1, 2026. Interested? Apply on the company site.
Town logo

About Town

Town | The AI Assistant That Does the Work

Town, the AI assistant that manages your inbox, schedules meetings, and handles busywork — working quietly in the background for you.

AI AssistantInboxesSchedulesAutomationIntegrationsTownieTowniesProductivity
Company profileWebsite

More open roles at Town