← All roles

Staff Software Engineer, Frontier Security Team

Snowflake Inc.· Engineering
Apply on company site
US-CA-Menlo Park$236k - $295kFull-timePosted 4w ago

About the role

Staff Software Engineer role on Snowflake's Frontier Security AI team, leading design and development of Agentic Harness and agent evaluation platform to deliver production-grade LLM applications, intelligent agents, AI infrastructure, and evaluation systems for enterprise customers.

Responsibilities

  • Architect and build the Agentic Harness that executes complex, multi-step AI workflows across models, tools, data, and services.
  • Design stable interfaces for tool execution, context construction, state management, memory, permissions, retries, fallbacks, and human review.
  • Own agent quality end to end by building evaluation harnesses, representative datasets, automated graders, experiment pipelines, and release gates.
  • Convert ambiguous reports such as "the agent feels worse" into measurable failure modes, reproducible tests, and durable fixes.
  • Analyze production agent trajectories to identify failures in reasoning, retrieval, tool use, context, orchestration, and application code.
  • Close the loop between production incidents, root-cause analysis, evaluation coverage, and regression prevention.
  • Develop offline and online measurements for task completion, correctness, groundedness, safety, latency, reliability, and cost.
  • Build simulation and replay infrastructure for golden-set tests, adversarial scenarios, model comparisons, and large-scale experiments.
  • Improve agent efficiency through model routing, prompt and semantic caching, context compaction, tool-result management, and token optimization.
  • Productionize new model capabilities as secure, observable, multi-tenant services with clear operational controls.
  • Establish standards for evaluation design, including sampling, ground-truth quality, grader calibration, leakage prevention, and statistical significance.
  • Define technical direction across multiple teams and lead projects whose scope extends beyond a single service.
  • Mentor engineers, raise the quality of architecture reviews, and remain directly involved in implementation and debugging.

Qualifications

  • 9+ years of software engineering experience, including technical leadership of complex production systems.
  • Direct experience shipping and operating LLM applications, AI agents, or model-backed workflows in production.
  • Strong background in distributed systems, service architecture, high-throughput APIs, concurrency, and failure handling.
  • Experience building an agent runtime, workflow engine, developer platform, evaluation system, or similar infrastructure.
  • Demonstrated ability to evaluate nondeterministic systems without relying on a single aggregate score.
  • Fluency in Python and strong proficiency in at least one systems or application language such as Java, Go, Rust, or TypeScript.
  • Hands-on knowledge of tool calling, structured generation, retrieval, context engineering, prompt management, and model APIs.
  • Experience with production observability, including structured traces, replay, metrics, logs, and incident diagnosis.
  • Ability to balance agent quality with latency, reliability, security, and inference cost.
  • Track record of setting technical direction and delivering results across organizational boundaries.
  • Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
  • Clear written and verbal communication with engineering, product, and leadership audiences.

Skills

Required:Software architecture and designDistributed systemsPythonSystems programming language experienceLLM applications / AI agentsObservability & incident diagnosisEvaluation designMentorship & leadership
Last updated Sep 24, 2026. Interested? Apply on the company site.

About Snowflake Inc.

The Data Platform Built for Data Teams.

Snowflake powers AI, data engineering, applications, and analytics on a trusted, scalable AI Data Cloud—eliminating silos and accelerating innovation.

AIData CloudData EngineeringAnalyticsApplications & Collaboration
Company profileWebsite

More open roles at Snowflake Inc.