ML Engineer — LLM Evaluation
San Francisco, CAFull-timePosted 11mo ago
About the role
ML Engineer role at Dynamo AI focusing on evaluation of LLMs with emphasis on safety, privacy, and responsible AI. Hybrid/onsite in San Francisco, full-time role within a research-oriented engineering team building a platform for private and compliant Gen AI models.
Responsibilities
- Own LLM evaluation processes and methods with a focus on generating benchmarks representative of real-world usage and safety vulnerabilities.
- Generate high quality synthetic data, curate labels, and conduct rigorous benchmarking.
- Deliver robust, scalable, and reproducible production code.
- Push the envelope by developing methods for benchmarking that revamps how we assess the best LLMs for harmlessness and helpfulness. Your research will directly empower our customers to more feasibly deploy safe and responsible LLMs.
- Co-author papers, patents, and presentations with our research team by integrating other members’ work with your vertical.
Qualifications
- Domain knowledge in LLM evaluation and data curation techniques.
- Extensive experience in designing and implementing LLM benchmarking, extending previous methods. Comfortability with leading end-to-end projects.
- Adaptability and flexibility. In both the academic and startup world, a new finding in the community may necessitate an abrupt shift in focus. You must be able to learn, implement, and extend state-of-the-art research.
- Preferred: past research or projects in benchmarking LLMs.
Skills
Required:LLM evaluationLLM benchmarkingData curationProduction code
Nice to have:Synthetic data generationResearch collaboration
Last updated Oct 7, 2026. Interested? Apply on the company site.
About Dynamo AI
Compliant-Ready AI for the Enterprise
End-to-end privacy, security, and compliance solutions to prepare your organization for emerging AI regulations.
machine-learningprivacydata-engineering
More open roles at Dynamo AI
- Product Manager, AI Governance & Compliance2w agoHybrid (San Francisco, CA)
- Product Manager, AgentWarden (Agent Security)2w agoHybrid (San Francisco, CA)
- Product Analyst3mo agoHybrid (San Francisco, CA) · Product
- ML Research Scientist Intern – Dynamo Guard / Dynamo Eval / AgentWarden8mo agoSan Francisco, CA, New York, NY +2 more · Engineering
- ML Research Engineer Intern – Dynamo Guard / Dynamo Eval / AgentWarden8mo agoHybrid (New York, NY; San Francisco, CA; Remote)