← All roles

Staff Software Engineer (Agentic Search) - Crawler

Nebius· Agentic Search
Apply on company site
Londonfull-timePosted 5d ago

About the role

Staff Software Engineer focused on content acquisition and crawling infrastructure for an agent-native search platform. Design and operate web-scale crawling, ingestion, and content processing to support real-time, scalable AI agent data retrieval and ranking.

Responsibilities

  • Design, implement, and operate web-scale crawling systems for acquiring content from the internet
  • Build ingestion workflows for internal and external data sources, including crawlers, structured feeds, and partner integrations
  • Develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies
  • Build systems for URL discovery, deduplication, content extraction, and crawl orchestration
  • Ensure reliable operation of crawling infrastructure under high-throughput conditions
  • Define observability and quality metrics for crawl coverage, freshness, throughput, and content quality
  • Monitor resource usage, bandwidth consumption, and infrastructure cost
  • Collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements
  • Enable safe experimentation with crawling strategies and content acquisition policies

Qualifications

  • 8+ years of experience building backend or distributed systems
  • Strong Go or C++ expertise
  • Experience with large-scale distributed systems (10k+ RPS, billions of URLs, high-throughput pipelines)
  • Understanding of web protocols (HTTP, DNS, TLS), crawling, scraping, and content extraction
  • Experience operating production systems and debugging failures in distributed environments
  • Strong understanding of scalability, fault tolerance, and resource management

Skills

Required:Backend engineeringGo or C++Distributed systemsWeb protocols

Benefits

  • Competitive compensation
  • Career growth and learning opportunities
  • Flexibility and ownership
  • Collaborative and innovative culture
  • Opportunity to work on impactful AI projects
  • International environment and talented teams
Last updated Oct 21, 2026. Interested? Apply on the company site.
Nebius logo

About Nebius

The ultimate cloud for AI explorers

Discover the most efficient way to build, tune and run your AI models and applications on top-notch NVIDIA® GPUs.

AI CloudToken FactoryGPUKubernetesSlurmManaged servicesTerraformAPI
Company profileWebsite

More open roles at Nebius