← All roles

Research Engineer - Benchmarks

Firecrawl· Engineering
Apply on company site
Hybrid (San Francisco, CA / Toronto, ON)$250k - $290kFull-timePosted 3w ago

About the role

Role focused on designing, running, and shipping independent, public benchmarks of data providers for Alexandria, including datasets, ground truth, scoring, and automated harnesses. You own the end-to-end benchmark workflow and publish weekly results that inform provider selection within Alexandria and beyond.

Responsibilities

  • Design and run head-to-head benchmarks of data providers against verified ground truth.
  • Build and maintain test datasets and ground truth, preventing staleness or leakage.
  • Own the automated harness and the weekly benchmark release; ensure reproducibility, versioning, and defensibility of each run.
  • Measure metrics buyers care about: accuracy, coverage, freshness, speed, and cost.
  • Collaborate with marketing to publish public leaderboard pages.
  • Integrate benchmark outcomes into Alexandria to influence provider calls.
  • Expand benchmark categories within Alexandria and beyond.

Qualifications

  • Shipped evals or benchmarks with trustworthy results and clear methodology.
  • Experience in frontier labs, data companies, public benchmark projects, or OSS benchmarks is valued.
  • Strong Python and API integration skills; comfortable with messy vendor APIs, rate limits, and inconsistent schemas.
  • Experience building test sets and scoring methods (sampling, labeling, inter-rater agreement, trusted LV/LLM judgments).
  • Proactive use of AI to improve workflows and shipping results without waiting for instructions.
  • Clear written communication of findings for engineers, marketers, and vendors.

Skills

Required:PythonAPI IntegrationsBenchmarking Methodology

Benefits

  • Salary: $250,000–$290,000 USD/year (SF) / $210,000–$224,000 CAD/year (Toronto).
  • Competitive equity.
  • Generous PTO (15 days, up to 24+ days as needed).
  • 12 weeks fully paid parental leave.
  • Wellness stipend ($100/month).
  • Learning & Development allowance ($1,000/year).
  • Team offsites and sabbatical after 4 years (3 paid months).
Last updated Oct 21, 2026. Interested? Apply on the company site.
Firecrawl logo

About Firecrawl

The web data API for AI

Firecrawl is a web data infrastructure platform that converts websites into clean, structured data optimized for AI applications through a simple API. Originally built as an internal tool at Mendable to solve their own data ingestion challenges, the team recognized the broader need for reliable web data extraction and transformed it into a standalone product. The platform combines web search, scraping, and AI-powered parsing into a unified solution, focusing on the core developer priorities of reliability, speed, and ease of use. By abstracting away the complexities of web data extraction, Firecrawl serves as the foundational data layer that enables AI applications to seamlessly access and process web content in the exact format they need.

artificial-intelligencebrex-fastest-growing-2025developer-toolsopen-source
Company profileWebsite

More open roles at Firecrawl