Member of Technical Staff, Evaluations
New York, NY$175k - $250kFull-timePosted 1mo ago
About the role
Hardware engineering data lives behind the walls of the organizations building frontier devices. The role focuses on evaluating models and harnesses across electrical engineering and hardware engineering tasks, building measurement disciplines, automated pipelines, analyses, and verifiable scoring systems, reporting to the CTO.
Responsibilities
- Harden an automated reporting pipeline
- Design an ablation to understand why a harness change caused a regression
- Analyze agent trajectories to find where a model loses efficacy on a dense datasheet and convert the problem into a measurable, tracked phenomenon
- Work with an electrical engineer to build synthetic datasets that push models and harnesses to their limits
- Design and calibrate an LLM judge for a task where ground truth is expensive, and prove it agrees with expert engineers well enough to rely on
- Own the design and reliability of evaluation harnesses and build automated, regular reporting pipelines
- Make regressions impossible to miss and results legible to engineers, leadership, and partners
- Design ablations that isolate the effect of individual changes to models, harnesses, tools, and context, and attribute contributors to performance/cost/runtime
- Build general analyses over agent trajectories that surface why the agent fails and identify performance clusters and root causes
- Leverage agent trajectories to build improvement loops for skills and harnesses across engineering tasks
- Craft judging and scoring systems that are verifiable wherever possible, and calibrate LLMs used as judges
- Partner with Operations and Electrical Engineering to hand-craft synthetic datasets that stress capabilities and cover long tail of real engineering work
- Interface with technical counterparts at partner organizations and customers, translating their notions of good into measurable artifacts
- Set the standard for evaluation rigor as capabilities, tools, and domains expand
Qualifications
- Strong Python skills, including building research or production infrastructure, data pipelines, and tooling scalable
- Demonstrated experience designing evaluations, benchmarks, or metrics for ML systems, ideally for LLMs and/or agentic systems
- Hands-on experience with LLM agents: harnesses, tool use, and analysis of agent trajectories
- Strong grounding in experimental design and statistics, and sound judgment about metrics trustworthiness
- Excellent written and verbal communication, especially explaining technical results to specialists and non-specialists
- Bias toward automation and building pipelines to avoid repetitive manual work
- Background or working fluency in electrical/hardware engineering (preferred) or ability to ramp quickly on technical domains
- Proficiency in calibrating and validating LLM-as-judge systems (preferred)
- Experience sourcing/curating synthetic datasets (preferred)
- Expertise in evaluating multimodal/spatiotemporal reasoning (preferred)
- Track record of building dashboards and trusted reporting (preferred)
- Experience working with customers or external partners (preferred)
Skills
Required:PythonEvaluation designLLM agentsExperimental design & statisticsCommunicationAutomation
Benefits
- 100% of monthly premiums covered with Aetna medical vision, and dental insurance for you and your dependents
- 401(k) Retirement Plan
- Unlimited PTO
- Lunch every day from local restaurants via Sharebite
- Relocation support provided
Last updated Oct 22, 2026. Interested? Apply on the company site.
About Arena Physica
Electromagnetic Superintelligence
HomeThesisPlatformServicesResearchCompanyCareersNews
More open roles at Arena Physica
- Member of Technical Staff, Product2mo agoNew York · Engineering
- Member of Technical Staff, Data & Training Infrastructure3mo agoNew York, NY · Engineering
- Research Scientist, Machine Learning3mo agoHybrid (New York, NY; SF Bay Area) · Applied Research