AI/ML Engineer
About the role
Production monitoring for AI agents. Role focuses on detecting silent failures in production, building evals, and ensuring reliable AI agent behavior.
Responsibilities
- Own detection quality. Find the failures that don't look like failures: compliant but wrong, omissions, and patterns that only show up across thousands of traces
- Turn implicit signals into evidence. Rephrasing, abandonment, retries, and the other ways users tell you something broke without saying so
- Build the evals for our own system. If we can't measure precision on a problem with no labels, we can't improve it
- Make patch generation trustworthy. Reproduce the failure, verify the fix, and know when to stay quiet instead of opening a bad PR
- Keep it affordable. LLM-as-judge on every event is easy. Doing it at a cost per event that doesn't eat the business is the actual job
- Read real customer traces every week. The best ideas here come from staring at production, not papers
Qualifications
- High slope over years of experience. New grads and dropouts welcome
- A track record of shipping things people actually use
- Hands-on with LLMs in production: evals, LLM-as-judge, embeddings, and knowing when a smaller model or no model is the right call
- Real research taste. You can tell a real improvement from noise, even when there's no ground truth to check against
Skills
About Lemma
Continuous learning for AI agents
Lemma enables AI agents to continuously improve from user feedback and production outcomes. Instead of relying on manual prompt tuning and reactive debugging, Lemma detects when an agent’s performance drifts, pinpoints the exact step causing failures, generates optimized prompt candidates, and automatically delivers improvements through our API or by opening a pull request in your codebase. Built by ex-AI engineers who felt this pain firsthand, Lemma turns static agents into self-improving systems, helping teams ship reliable AI products faster with compounding performance gains over time. Deploy 1x. Learn forever.
More open roles at Lemma
- Product Engineer3w agoSan Francisco, CA · Engineering