Software Engineer- Benchmarking
San Francisco, NYC, or Fully Remote$160k - $210kFull-timePosted 2mo ago
About the role
Software Engineer to build and run the platform behind AI model evaluations, collaborating with researchers to prepare benchmark datasets, build evaluation pipelines and environments, and publish results.
Responsibilities
- Prepare and maintain benchmark datasets: cleaning, preparation, conversion into runnable formats, ongoing maintenance, validate completeness and consistency, flag ambiguities.
- Build and maintain evaluation pipelines: infrastructure that runs evaluations consistently across model APIs and terminal agents.
- Build evaluation environments: create lightweight, containerized environments and viewers for tasking and evaluating model performance on tool use.
- Support model experiments: help fine-tune small open-source LLMs and compare baseline against post-training performance.
- Develop the scoreboard and leaderboard: publish views of results at model, benchmark, task, domain, and rubric levels.
- Build analysis tools: identify recurring failure modes, fairly compare models/agent scaffolds, track capability changes over time.
- Collaborate: work with researchers and engineers to ensure data and outputs are accurate and well integrated into publications.
- Build tooling for data creation and review: lightweight HTML viewers and internal tools for task authoring, review, quality control, and structured data collection.
Qualifications
- 4+ years of professional experience building and maintaining complex systems, strong Python
- Data rigor in preparing, cleaning, and maintaining datasets with care for comparability of results
- Experience with containers and environments (Docker) and building reproducible execution environments
- Collaborative: able to work with researchers and scientists to translate methodology into working systems
- Hands-on experience running AI evaluations or with Harbor, Terminal-Bench, or Inspect is a strong plus
Skills
Required:PythonData preparation and cleaningDocker
Benefits
- Competitive salary and equity
- Medical, dental, and vision coverage
- 401(k)
- Monthly wellness and fitness stipend
- Paid time off policy, plus company holidays
- Annual company off-sites (Tahoe, Mendocino, Mexico City, San Diego, Park City)
- Flexible remote work policy
- Paid parental leave (12 weeks fully paid bonding leave; birthing parents 8–10 additional weeks of paid recovery)
Last updated Oct 23, 2026. Interested? Apply on the company site.
About Office Hours
Office Hours: Get paid for your expertise
Office Hours connects your expertise to paid opportunities—calls, surveys, and expert data work. We handle scheduling, compliance, and payments.
IndustryCompanyRoleSpecialtyLocation
More open roles at Office Hours
- Senior Frontend Engineer1w agoSan Francisco, NYC, or Fully Remote · Product
- Client Solutions Manager2w agoHybrid (New York City) · Client Solutions
- Business Operations Associate2w agoHybrid (San Francisco) · Business Operations
- People Operations Lead4w agoHybrid (San Francisco, NYC) · People
- IT Operations Lead1mo agoHybrid (San Francisco) · People