← All roles

Applied AI Engineer

Monte Carlo· SPEDD (Security, Product, Engineering, Design, Data)
Apply on company site
Remote$200k - $260k + equityFull-timePosted 1mo ago

About the role

Role focused on end-to-end development of agent-powered features and production-grade evaluation/monitoring within Monte Carlo's agent observability platform. Build from ambiguous problems through research, prototyping, and production, collaborating with engineering, data science, and PM teams.

Responsibilities

  • Take an open problem end-to-end — from research and prototyping through production, killing what doesn't work before it becomes someone's roadmap
  • Design and ship agent-powered features — root-cause analysis, incident triage, monitor generation — and integrate them into the platform with our engineering team
  • Build the eval infrastructure that makes those features safe to change: golden datasets, regression suites, offline and online scoring, and the judgment calls about what 'good' means
  • Own retrieval and context pipelines over customer metadata, lineage, and query history, and instrument agent behavior in production — traces, failure taxonomies, cost and latency budgets — to close the loop on quality
  • Partner with data science on detection quality and experiment design, and with PM on what an agent should do versus what it merely can do
  • Set the technical bar for how we build with LLMs — patterns, guardrails, and the internal tooling other engineers reuse

Qualifications

  • You've built agents in production with real users. Not integrated a framework. Built them — agents with real autonomy and internal loops, where the model uses tools and decides what to do next without a human in the middle, and you kept them running once real users showed up.
  • You've run evals and monitored agents after launch. You own an eval framework — golden datasets, regression suites, offline and online scoring — and have observed agents in production.
  • Python, plus an ML or data science background. Proficiency in backend; solid understanding of models and their behavior.
  • 3+ years in ML, data science, or software engineering — at companies that ship quickly (startups, AI-native teams, or high-growth product companies).
  • You work from a problem, not a spec. You design experiments, build minimal viable versions, and take successful approaches into production.
  • You use AI tools daily (Claude or equivalents) as part of coding and research; this is backend/model-layer work, not frontend.
  • You'd rather ship than polish — focus on quickly delivering working solutions; can discern when deeper exploration is warranted.

Skills

Required:PythonML / Data ScienceProduction-grade ML systemsExperiment design / A/B testing

Benefits

  • Stock Options
  • Healthcare plans
  • 401k Retirement Plan
  • Wellness Stipend
  • Home Office Stipend
  • Cell Phone or WIFI reimbursement
  • Paid Parental Leave
  • Flexible Time Off
  • Generous Travel Policy
Last updated Oct 1, 2026. Interested? Apply on the company site.

About Monte Carlo

The only end-to-end data and AI observability platform for enterprise teams.

Go beyond data quality to unlock true AI observability with the only end-to-end data and AI observability platform for enterprise teams.

data qualityAI observabilitydata governancemonitoringlineageperformancealertingmetadata
Company profileWebsite

More open roles at Monte Carlo