Machine Learning Engineer, Reliability
fal.ai· Engineering
RemoteFull-timePosted 3mo ago
About the role
Hybrid ML Engineering / Site Reliability Engineering role focusing on reliability, security, and safety of fal's fleet of generative media model APIs, ensuring uptime, performance, and safe operation at scale.
Responsibilities
- Own availability, latency, and throughput SLOs across a large fleet of generative media model APIs serving production traffic at scale
- Build the monitoring, alerting, and observability needed to catch ML-specific failures, output quality degradation, pipeline breakage, model regressions before customers do
- Harden model deployment workflows with canary releases, shadow testing, automated rollbacks, and validation gates so new model versions ship safely
- Drive the security posture of the model fleet: secure model serving, abuse and misuse detection, rate limiting, and protection against adversarial usage patterns
- Operationalize safety systems for generative media, content moderation pipelines, safety classifiers, and guardrails that run reliably at inference time without compromising performance
- Lead incident response for model API outages and degradations, run postmortems, and drive the engineering work that prevents recurrence
- Improve capacity planning, autoscaling, and GPU fleet efficiency for inference workloads under highly variable traffic
- Partner with model and infrastructure teams to make reliability, security, and safety requirements part of how new models get onboarded to the platform
- You will have access to our massive GPU cluster for inference and evaluation
Qualifications
- 5+ years of professional experience, with 2 year experience operating production ML or high-scale API systems, ideally with on-call ownership
- Experience working with and supporting diffusion models in production
- Strong systems fundamentals: distributed systems, networking, observability, and incident management
- Working knowledge of modern generative models (diffusion, transformers) and their failure modes in production
- Familiarity with security and safety practices for ML systems ,abuse prevention, content safety, or trust & safety engineering experience is a strong plus
- A bias toward automation, measurement, and blameless postmortems
- Core technologies we use include Python, torch, diffusers, Kubernetes, and the fal Python SDK
- You'll work alongside a team dedicated to quickly iterating on and deploying new AI breakthroughs — your job is to make sure that speed never comes at the cost of reliability
Skills
Required:Availability managementObservability and monitoringModel deployment workflowsSecurity and safety
Nice to have:Safety systems for generative media
Benefits
- Interesting and challenging work
- Competitive salary and equity
- A lot of learning and growth opportunities
- Regular team events and offsites
Last updated Oct 24, 2026. Interested? Apply on the company site.
About fal.ai
ai-infrastructure-and-developmentwing-list-2026wing-list-2026-late-stage
More open roles at fal.ai
- Account Executive, Commercial2d agoSF Office · Sales
- Marketing Operations2d agoSan Francisco · Marketing
- AI Native Account Executive, Compute2d agoRemote · Sales
- Compute Deals1w agoSan Francisco · Operations
- Infrastructure Accounting Lead1w agoSan Francisco · Finance