← All roles

Member of Technical Staff, AI Evaluation

P-1 AI· AI Engineering
Apply on company site
Remote$200k - $300kFull-timePosted 2w ago

About the role

Role focused on evaluating AI for physical engineering through developing evaluation methodologies, automated judges, and human evaluations; works with customers, SMEs, partners, and internal teams to ensure evaluations reflect real engineering workflows and guide product development.

Responsibilities

  • Study how engineers and customers use Archie in the field, identify where Archie succeeds or fails from an engineering perspective, and turn those observations into actionable evaluations.
  • Recreate real-world engineering tasks and failure modes in environments that can be repeatedly used by our development teams.
  • Build and refine evaluation methodologies for Archie, including automated judges and human evaluation processes.
  • Assess whether automated judges actually align with expert human judgment and improve them when they do not.
  • Work across engineering, product, customers, SMEs, and external partners to make sure our evaluations reflect real engineering workflows rather than abstract benchmarks.
  • Use evaluation results to help the team understand where Archie needs to improve and which problems matter most to users.

Qualifications

  • Technical background in a physical engineering domain such as power systems, power electronics, mechanical engineering, aerospace, robotics, or a closely related field.
  • Strong understanding of agentic AI systems to design and reason about evaluations.
  • Experience with AI evaluations, including creating evaluation tasks, designing or crafting judges, and comparing automated evaluation against human judgment.
  • Comfortable working directly with customers, engineers, subject matter experts, partners, and internal technical teams to extract what matters from ambiguous workflows.
  • Strong product judgment, with the ability to turn observations about how a system is being used or failing into concrete evaluation work.
  • Motivated by building superintelligence for engineering and excited by the challenge of making AI systems genuinely capable in the physical world.

Skills

Required:Technical background in physical engineeringAgentic AI systems understandingAI evaluations experienceStakeholder collaborationProduct judgmentMotivation for AI in engineering

Benefits

  • Competitive salary
  • meaningful equity ownership
  • healthcare
  • dental
  • vision
  • 401(k) match
  • unlimited PTO
Last updated Oct 12, 2026. Interested? Apply on the company site.

About P-1 AI

We are building an AI engineer agent for the physical world. His name is Archie. We try to maximize Archie’s anthropomorphism so that he fits seamlessly into existing engineering teams and workflows.

AI engineer agentmultiphysicsdesign representationengineering tasksdata center engineeringcoolingcritical power systemsautomotive
Company profileWebsite

More open roles at P-1 AI