← All roles

Agent Evaluation and Self-Improvement Engineer

NVIDIA· Engineer, Sys SW
Apply on company site
Santa Clara, CA$224k - $356.5kfull-timePosted 4d ago

About the role

Principal engineer to define evaluation strategy across agent systems, harnesses, and products; design self-improvement loops for agents to autonomously propose and test changes with measurable gains and safe human oversight.

Responsibilities

  • Build evaluation that works the same across harnesses, with automated checks on every change that catch regressions before users do.
  • Mine production traces for failures, usage patterns, missing skills/tools, cost and latency hotspots; turn them into eval cases and improvement signals.
  • Evaluate the eval datasets themselves: coverage of real usage, duplicate cases, and how closely they match production.
  • Build the self-improvement loop for evolving harnesses: agents propose changes to their own skills, plugins, and harness configuration, and sandboxed evals test each one; only measurable gains ship with human sign-off and limits agents can't change.
  • Create realistic enterprise environments: sandboxed replicas of issue trackers, code repositories, chat, and internal APIs, with synthetic/anonymized data for end-to-end tasks.
  • Measure multi-agent swarms as a team: task division, handoffs, permission checks, task duration, and token usage.
  • Keep evals honest: calibrate model/agent judges against human labels; detect and mitigate cheating; collaborate with product, platform, and hardware teams; help shape industry evaluation standards and benchmarks.

Qualifications

  • BS or MS in Computer Science, Engineering, or related field (or equivalent experience)
  • 12+ years building large-scale systems, including evaluation systems for ML or agents and data pipelines
  • Hands-on experience building and evaluating agents on an open or closed agent harness
  • Solid grounding in experiment design: statistical significance, variance across repeated runs, and A/B testing
  • Experience setting technical direction across teams as a senior individual contributor
  • Proficiency in Python, Go, Rust, or similar

Skills

Required:Large-scale systems design and evaluation pipelinesExperiment design and A/B testingAgent/harness developmentPython, Go, Rust

Benefits

  • equity
  • benefits (see NVIDIA benefits)
Last updated Oct 23, 2026. Interested? Apply on the company site.

About NVIDIA

World Leader in AI Computing

NVIDIA invents the GPU and drives advances in AI, HPC, gaming, creative design, autonomous vehicles, and robotics.

AIautonomous vehiclesgaminggpugraphics cardhigh performance computingliquid-ai-recruiting-targetllm
Company profileWebsite

More open roles at NVIDIA