Member of Technical Staff (Evals)
About the role
Role focused on measuring AI Employee performance and building evaluation systems for Jarmin’s AI products. Involves defining metrics, constructing evaluation pipelines, and collaborating with researchers to ensure production-grade evaluations.
Responsibilities
- Own how we measure AI Employee performance on the job.
- Evaluation systems: datasets, offline replay, scorers, and regression alerts
- Feedback loops for models and agents
- Autonomous failure analysis
- Defining what good, degraded, and failed runs look like, and alerting on them
Qualifications
- Experience with built evaluation or measurement systems (e.g., AI evals, experimentation, ranking, or search quality)
- Ability to translate ambiguous quality questions into concrete metrics and decisions
- Strong software engineering fundamentals and experience shipping production systems
- Comfort operating where patterns aren’t fully established
- Willingness to work in person in San Francisco (preferred)
Skills
About Jarmin
24/7 Machine Learning Engineer employees
Jarmin.ai is your 24/7 ML eng employee that you hire. Just talk to Jarmin like any other employee and Jarmin handles your AI/ML work from there. Completely hand over full initiatives or individual tasks. Jarmin’s founding team is from Meta SuperIntelligence Labs (Staff Engineers), Apple (Machine Learning Engineer), AWS (scaling infra), Lockheed-Martin (Research), and JPMorganChase (SWE) with experience building AI/ML and automating complex work of Researchers, Data Scientists, ML Engineers with AI.
More open roles at Jarmin
- Founding Sales - Account Executive1mo agoHybrid (San Francisco, CA, US)
- Member of Technical Staff1mo agoSan Francisco, CA · Engineering
- Member of Technical Staff (Models)2mo agoHybrid (San Francisco, CA) · Engineering
- Member of Technical Staff (Agents)2mo agoHybrid (San Francisco) · Engineering
- Member of Technical Staff (Product)2mo agoHybrid (San Francisco, CA) · Engineering