← All roles

Software Engineer, Web Crawling

Exa· Engineering
Apply on company site
San Francisco, California$180k - $350kFull-timePosted 11mo ago

About the role

Software Engineer focused on building and scaling web crawlers for large-scale web data processing, including design of distributed crawling systems, handling dynamic content, and optimizing crawl efficiency.

Responsibilities

  • Build a distributed crawler that can handle 100M+ pages per day
  • Optimize crawl politeness and rate limiting across thousands of domains
  • Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
  • Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency

Qualifications

  • Extensive experience building and scaling web crawlers, or demonstrated ability to ramp up quickly
  • Experience with high performance language (C++, Rust, etc.)
  • Familiarity with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol)
  • Comfortable optimizing a system to an exceptional degree
  • Interest in finding high quality knowledge and its global impact

Skills

Required:Web crawlingHigh-performance programming
Nice to have:TypeScriptPlaywrightChrome DevTools Protocol (CDP)

Benefits

  • Premium healthcare benefits (medical, dental, vision)
  • Fertility benefits
  • 16 weeks of fully paid parental leave for all new parents
  • Monthly wellness stipend
Last updated Sep 16, 2026. Interested? Apply on the company site.
Exa logo

About Exa

Web search rebuilt for LLMs

Exa is a search engine built for AIs. We enable developers to filter the internet's content with complex queries in order to bring the world's knowledge into their AI apps.

ai-infrastructure-and-developmentapisartificial-intelligencebrex-fastest-growing-2025searchwing-list-2026wing-list-2026-mid-stage
Company profileWebsite

More open roles at Exa