← All roles

Staff Software Engineer (Agentic Search) - Crawler

Nebius· Agentic Search
Apply on company site
ZurichFull-timePosted 5d ago

About the role

Staff Software Engineer to build and operate web-scale crawling and content ingestion infrastructure for an agent-native search platform focused on AI consumption. Responsible for designing distributed crawling, scheduling, ingestion, and observability to support real-time retrieval and ranking of large-scale web content.

Responsibilities

  • Design, implement, and operate web-scale crawling systems for acquiring content from the internet
  • Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations
  • Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies
  • Build systems for URL discovery, deduplication, content extraction, and crawl orchestration
  • Ensure reliable operation of crawling infrastructure under high-throughput conditions
  • Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality
  • Monitor resource usage, bandwidth consumption, and infrastructure cost
  • Collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements
  • Enable safe experimentation with crawling strategies and content acquisition policies

Qualifications

  • 8+ years of experience building backend or distributed systems
  • Strong Go or C++ expertise
  • Experience with large-scale distributed systems (10k+ RPS, billions of URLs, high-throughput pipelines)
  • Understanding of web protocols (HTTP, DNS, TLS), crawling, scraping, and content extraction
  • Experience operating production systems and debugging failures in distributed environments
  • Strong understanding of scalability, fault tolerance, and resource management

Skills

Required:Backend or distributed systemsGo or C++Large-scale distributed systemsWeb protocols, crawling, content extractionProduction systems operation and debuggingObservability and metrics

Benefits

  • Competitive compensation
  • Career growth and learning opportunities
  • Flexibility and ownership
  • Collaborative and innovative culture
  • Opportunity to work on impactful AI projects
  • International environment and talented teams
Last updated Oct 21, 2026. Interested? Apply on the company site.
Nebius logo

About Nebius

The ultimate cloud for AI explorers

Discover the most efficient way to build, tune and run your AI models and applications on top-notch NVIDIA® GPUs.

AI CloudToken FactoryGPUKubernetesSlurmManaged servicesTerraformAPI
Company profileWebsite

More open roles at Nebius