← All roles

Forward Deployed Engineer - Language Models

Artificial Analysis· Platform & Forward Deployed Engineering
Apply on company site
San Francisco (On-site), Melbourne (On-site)Full-timePosted 1mo ago

About the role

Forward Deployed Engineer responsible for day-to-day operation of the language model benchmarking stack, onboarding models, running benchmarks, and acting as the primary technical liaison to AI lab customers.

Responsibilities

  • Operate and maintain our Python-based language model benchmarking pipeline end-to-end: onboard new models, configure evaluations, execute benchmark runs, and validate results
  • Debug issues across the stack, from API endpoint timeouts and errors to unexpected benchmark outputs, and resolve them quickly
  • Serve as the primary technical contact for AI lab customers: communicate benchmarking results clearly, explain methodology, field technical questions, and troubleshoot integration issues via Slack and video conferencing
  • Monitor benchmark runs for anomalies, investigate discrepancies, and ensure the accuracy and integrity of published results
  • Maintain documentation of processes, known issues, and model-specific configurations
  • Collaborate with the engineering team to flag pipeline improvements and contribute to process refinements
  • Stay current with new model releases, API changes, and developments across the language model ecosystem

Qualifications

  • 3+ years of experience in a client-facing technical role - solutions engineering, support engineering, technical consulting, or similar
  • Strong Python proficiency and comfort working with complex codebases you didn’t write
  • Hands-on experience working with AI/ML model APIs (OpenAI, Anthropic, Google, Meta, etc.)
  • Excellent debugging skills; ability to trace issues across APIs, data pipelines, and code
  • Strong written and verbal English communication skills, with the ability to explain technical concepts clearly to technical stakeholders
  • Highly responsive and reliable; ownership of customer issues and follow-through
  • Comfortable with operational, repeatable work; focus on running things exceptionally well
  • High attention to detail and calm under pressure

Skills

Required:PythonAI/ML model APIsDebuggingCommunicationCustomer-facing

Benefits

  • Competitive compensation including equity
Last updated Oct 1, 2026. Interested? Apply on the company site.

About Artificial Analysis

Independent analysis of AI

AI ModelAPI ProvidersAnalysisBenchmarkingIndependenceEvaluationIntelligence IndexLeaderboards
Company profileWebsite

More open roles at Artificial Analysis