← All roles

Member of Technical Staff - Web Crawl Engineer

Reflection· Research
Apply on company site
San Francisco, CAFull-timePosted 4mo ago

About the role

Engineer responsible for building and operating large-scale web crawling systems to continuously discover, acquire, and process content from across the internet, enabling high-quality web data for frontier AI systems.

Responsibilities

  • Build and operate web-scale crawling infrastructure capable of continuously collecting data across billions of URLs
  • Design and optimize URL discovery, prioritization, scheduling, and crawl orchestration systems
  • Develop distributed crawlers that efficiently acquire content while respecting site constraints and operational requirements
  • Build systems for content extraction, rendering, parsing, and normalization across diverse web formats
  • Improve crawl coverage, freshness, efficiency, and quality through measurement and experimentation
  • Design infrastructure for large-scale recrawling, change detection, and incremental updates
  • Develop specialized crawlers for high-value domains, dynamic websites, and difficult-to-access content sources
  • Analyze crawl performance and web coverage to identify gaps, inefficiencies, and opportunities for improvement
  • Build observability, monitoring, and reliability systems for large-scale crawl operations
  • Debug production issues and continuously improve the performance, scalability, and resilience of crawling infrastructure

Qualifications

  • Experience building large-scale web crawling, search indexing, content acquisition, or internet-scale data collection systems
  • Strong understanding of crawling architectures, URL frontier management, scheduling, and distributed crawl coordination
  • Experience with large-scale distributed systems using technologies such as Ray, Spark, Beam, Flink, or similar frameworks
  • Familiarity with content extraction, HTML parsing, browser automation, rendering systems, and modern web technologies
  • Experience operating systems that process petabyte-scale datasets
  • Strong systems engineering skills, including reliability, observability, performance optimization, and debugging
  • Experience designing experiments and using data to improve crawl quality, coverage, and efficiency
  • Excellent communication skills and the ability to reason clearly about system tradeoffs and operational constraints

Skills

Required:Web-scale crawling design and implementationURL frontier management and crawl schedulingDistributed systems (Ray, Spark, Beam, Flink)Content extraction and renderingPetabyte-scale data processingReliability, observability, performance optimizationExperiment design and data-driven improvementCommunication and tradeoff reasoning

Benefits

  • Top-tier compensation: Salary and equity structured to recognize and retain our talent globally.
  • Stock options: Everyone who joins and contributes to Reflection's success gets to share in the upside through stock options.
  • Health & wellness: Comprehensive medical, dental, vision, and life, with an annual wellness allowance.
  • Meals: Lunch and dinner are provided in the office daily.
  • Life & family: 22 weeks paid parental leave for all new birthing and non-birthing parents, including adoptive and surrogate journeys.
  • Vacation days: Unlimited paid time off in the U.S. and 30 days in the U.K.
  • Sponsorship support: We sponsor visas to help exceptional talent join our team and support long-term immigration pathways where applicable.
  • Team building: We have regular off-sites, happy hours, and team celebrations.
Last updated Oct 1, 2026. Interested? Apply on the company site.

About Reflection

Building frontier open intelligence

Building frontier open intelligence.

liquid-ai-recruiting-target
Company profileWebsite

More open roles at Reflection