← All roles

Staff AI Engineer - Agent Observability

Datadog· Engineering
Apply on company site
Hybrid (Boston, MA; New York, NY)$234k - $300kFull-timePosted 6mo ago

About the role

Staff Engineer leading the development of features and foundational capabilities for Datadog’s LLM Observability product, focusing on monitoring, explaining, and improving AI systems in production with emphasis on LLMs and generative AI.

Responsibilities

  • Drive design and implementation of LLM observability features.
  • Ideate, prototype, and scale new product features to provide insights and drive improvements for generative AI systems
  • Work cross-functionally with other eng teams, product, UX, and applied science to iterate fast and find product-market fit
  • Develop and extend tools for tracing, evaluating, and debugging LLMs
  • Influence architecture decisions and mentor engineers to build resilient, high-performance systems
  • Stay close to customer pain points and use those insights to guide product and engineering priorities
  • Stay current with industry trends and advancements in machine learning and observability, driving innovation within the team

Qualifications

  • BS/MS/PhD in Computer Science, Engineering or related scientific field or equivalent experience
  • Deep understanding of distributed systems and scalable backend architectures
  • Hands-on experience building and shipping LLM-powered or GenAI applications
  • Understanding of model internals, inference pipelines, evaluation techniques, and prompt engineering
  • Ability to thrive in ambiguous, fast-changing spaces and have a product-oriented mindset
  • Excited to shape the next generation of AI observability tools from the ground up
  • Communicate clearly, think rigorously, and take pride in clean, maintainable code
  • Experience with observability tools/platforms

Skills

Required:LLM observability featuresProduct discovery and cross-functional collaborationArchitecture and systems design

Benefits

  • Competitive global benefits
  • Continuous professional development
  • Healthcare, dental, parental planning, mental health benefits, 401(k) plan and match, paid time off, fitness reimbursements, discounted employee stock purchase plan
Last updated Oct 22, 2026. Interested? Apply on the company site.

About Datadog

AI-Powered Observability and Security

See inside any stack, any app, at any scale, anywhere.

Company profileWebsite

More open roles at Datadog