← All roles

Evaluations - Member of Technical Staff

Simile· Research
Apply on company site
San Francisco, New York$200k - $400kFull-timePosted 2mo ago

About the role

Role focused on defining and executing evaluation methodologies for Simile's AI-based simulations of human behavior. Build measurement systems, run studies, and turn evidence into product and modeling priorities. Collaboration across model evals, statistics, behavioral science, and product to determine when simulations are reliable enough to inform real decisions.

Responsibilities

  • Build the measurement layer for behavioral simulation: design evals, metrics, rubrics, datasets, dashboards, and workflows that measure whether Simile’s models are accurately predicting human behavior across customer use cases, populations, question types, and decision contexts.
  • Partner with modeling to improve models: evaluate new model versions, diagnose regressions, identify priority areas for model-improvement cycles, and maintain stable eval suites that represent capabilities customers actually care about.
  • Contribute to product and applied evals: build evals for qualitative responses, retrieval, survey generation, AI-generated research reports, customer-facing outputs, and other product surfaces where model quality directly shapes customer trust.
  • Make ground truth and uncertainty legible: develop rigorous ways to compare simulated responses against human data, customer studies, Simile-collected ground truth, and behavioral datasets.
  • Automate evaluation workflows: build internal tools, scripts, dashboards, labeling workflows, validate evals, and turn fuzzy evaluation questions into working systems.
  • Expand what we can evaluate: develop repeatable benchmarks for behavioral predictions across transaction and product-interaction data, intervention effects, individual trajectories, and multi-agent settings.

Qualifications

  • Evaluation and Measurement Judgment: ability to turn ambiguous quality questions into validated measurements and justify decisions with baselines and rigorous testing.
  • LLM and Model Fluency: capacity to work with LLM outputs and understand modeling team needs; quick to learn new model concepts.
  • Statistical Judgment: comfort with noisy data, uncertainty, sampling, distributions, calibration, and bias.
  • Technical and Agentic Execution: skilled in building tools, dashboards, labeling workflows, analyses, and automated eval pipelines using Python, SQL, notebooks, and related tooling.
  • Hands-On Ownership and Collaboration: experience driving workstreams, building prototypes, inspecting data, debugging, and delivering usable results.

Skills

Required:Measurement designStatistical reasoningExperimentation and eval automationLLM/Model fluency

Benefits

  • Salary and equity package as described in compensation section.
  • Health & Wellness: medical, dental, and vision coverage.
  • Time Off: flexible time off policies.
Last updated Oct 26, 2026. Interested? Apply on the company site.

About Simile

Company profileWebsite

More open roles at Simile