Staff ML Engineer, Frontier AI
Ambience Healthcare· Engineering
San Francisco$244k - $305kFull-timePosted 7mo ago
About the role
Staff Machine Learning Engineer at Ambience responsible for setting technical direction of AI systems powering clinical products, driving production model improvements, post-training capabilities, and agentic AI architectures in a hybrid work setup.
Responsibilities
- Define the AI Quality and Evaluation Strategy: Establish how we measure production AI quality across LLM and agentic systems, including automated graders, regression testing, human evaluation, failure taxonomies, and offline-to-online validation.
- Drive Production Model Improvement: Identify high-impact failure modes across products and lead improvements across prompting, retrieval, context, routing, data, and post-training. Design experiments that determine the right intervention and prove that improvements hold in production.
- Advance Post-Training Capabilities: Shape how Ambience uses techniques such as supervised fine-tuning, preference optimization, reinforcement learning, distillation, and synthetic data to improve model behavior.
- Architect Agentic AI Systems: Set technical direction for production systems involving tool use, retrieval, context and state management, routing, orchestration, tracing, and failure recovery.
- Build Self-Improving AI Loops: Design the systems that turn production failures and user feedback into better datasets, evaluations, and model behavior, creating scalable improvement flywheels across products.
- Shape the AI Roadmap: Co-develop the 12-month technical roadmap, balancing near-term product wins with longer-term research and platform investments.
- Raise the Technical Bar: Lead technical reviews, mentor engineers, establish best practices, and create reusable systems and frameworks that improve the effectiveness of the broader AI team.
- Stay at the Cutting Edge: Distill insights from recent research in LLMs, agents, post-training, NLP, speech, and multimodal AI, and drive experiments that keep Ambience at the forefront of clinical AI.
Qualifications
- 7+ years in production ML, research engineering, or applied AI.
- Deep expertise with modern LLMs, transformers, and complex production AI systems.
- Experience designing evaluations for LLMs, agents, or other complex AI systems.
- Experience building or architecting production systems involving multiple models, tools, retrieval, context and state management, routing, orchestration, tracing, and failure recovery.
- Proficient in Python and modern ML frameworks; PyTorch preferred.
- Experience with production failures, user feedback, and active learning to improve model and system quality.
- Strong communication and collaboration skills with clinicians, product managers, researchers, and engineers.
- Passion for healthcare or mission-driven industries.
Skills
Required:Production AI systems designEvaluation and metrics for AI
Benefits
- Comprehensive medical, dental, and vision coverage for you and your dependents
- 401(k) with a company match of up to 3% of base salary
- Remote-friendly culture (San Francisco HQ) and full equipment provisioning
- Parental leave
- Annual company-wide and team off-sites, regular team lunches and all-hands with travel, lodging, and meals covered
- Flexible time off with no annual cap, company-wide holidays and an annual holiday shutdown Dec 24–Jan 1
Last updated Sep 23, 2026. Interested? Apply on the company site.
About Ambience Healthcare
The AI Platform Clinicians Choose for Documentation and Coding
The AI Platform Clinicians Choose for Documentation and Coding
AI PlatformDocumentationCodingHealthcareHealth System
More open roles at Ambience Healthcare
- Senior Product Marketing Manager (Chorus)2w agoRemote · Marketing
- Senior Manager, Growth Marketing3w agoRemote · Marketing
- Senior GRC Analyst3w agoHybrid (San Francisco) · Engineering
- Senior Machine Learning Engineer2mo agoSan Francisco · Engineering
- Product Lead, Inpatient2mo agoSan Francisco · Product & Design