← All roles

Senior Machine Learning Engineer

Ambience Healthcare· Engineering
Apply on company site
San Francisco$204k - $255kFull-timePosted 2mo ago

About the role

Senior Machine Learning Engineer role at Ambience building production-grade AI systems for clinical workflows, focusing on evaluation, agentic systems, data improvement flywheels, and end-to-end ownership of AI solutions in healthcare.

Responsibilities

  • Build Trustworthy AI Evaluation Systems: Design and own evaluation pipelines for LLM and agentic systems, combining automated graders, regression testing, production feedback, and human evaluation to measure real product quality.
  • Improve Production Model Behavior: Diagnose high-impact failure modes and test improvements across prompting, retrieval, context, routing, data, fine-tuning, or other model and system interventions.
  • Build Agentic AI Systems: Develop production systems involving tool use, retrieval, context and state management, routing, orchestration, tracing, and failure recovery.
  • Build Data and Improvement Flywheels: Turn production failures and user feedback into better datasets, evaluations, and model behavior through active learning and systematic iteration.
  • Stay at the Cutting Edge: Distill insights from recent research in LLMs, agents, NLP, speech, and multimodal AI and translate promising ideas into practical experiments.
  • Own AI Systems End-to-End: Work across models, data, evaluation, orchestration, serving, and observability, while remaining deeply hands-on in code and production debugging.

Qualifications

  • 5+ years in production ML, research engineering, or applied AI.
  • Built a consequential production AI system or materially improved model behavior in production.
  • Strong understanding of modern LLMs, transformers, and production AI systems.
  • Experience designing evaluations for LLMs, agents, or other complex AI systems.
  • Can turn ambiguous quality problems into measurable dimensions, datasets, and experiments.
  • Familiar with challenges such as grader bias, leakage, misleading aggregate metrics, regression detection, and offline-online mismatch.
  • Experience building production systems involving multiple models, tools, retrieval, context, state, routing, or orchestration.
  • Understand reliability and failure modes in complex AI workflows, not just individual model calls.
  • Proficient in Python and modern ML frameworks; PyTorch preferred.
  • Comfortable with deployment, observability, CI/CD, and containerized systems.
  • Hands-on: writes code, inspects traces, analyzes failures, and debugs production systems.
  • Skilled at building high-quality datasets and feedback loops.
  • Experienced using production failures, user feedback, and active learning to improve model and system quality.
  • Able to work closely with clinicians, product managers, and fellow engineers; strong communicator.

Skills

Required:Production AI engineeringEvaluation design for AIPythonCI/CD and containerization
Nice to have:PyTorch

Benefits

  • Comprehensive medical, dental, and vision coverage for you and your dependents
  • 401(k) with a company match of up to 3% of base salary
  • Remote-friendly culture (with a San Francisco HQ) and full equipment provisioning
  • Parental leave
  • Annual company-wide off-sites, team off-sites and regular team lunches and all-hands gatherings, with travel, lodging and meals covered
  • Flexible time off with no annual cap, company-wide holidays and an annual holiday shutdown from December 24–January 1
Last updated Sep 23, 2026. Interested? Apply on the company site.

About Ambience Healthcare

The AI Platform Clinicians Choose for Documentation and Coding

The AI Platform Clinicians Choose for Documentation and Coding

AI PlatformDocumentationCodingHealthcareHealth System
Company profileWebsite

More open roles at Ambience Healthcare