Research Engineer - Evaluations
Hybrid (New York, NY; SF Bay Area, CA; London, UK)$190k - $375kFull-timePosted 11mo ago
About the role
Role focused on designing and scaling infrastructure for model evaluation, building pipelines, metrics, and automated systems to close the loop between model output, evaluation, and improvement for multimodal generative AI.
Responsibilities
- Design and implement scalable pipelines for automated evaluation of generative models, with a focus on visual and multimodal outputs (image, video, text, audio).
- Develop novel metrics and evaluation models that capture fidelity, coherence, temporal consistency, and alignment with human intent.
- Integrate evaluation signals into training loops (including reinforcement learning and reward modeling) to continuously improve model performance.
- Build infrastructure for large-scale regression testing, benchmarking, and monitoring of multimodal generative models.
- Collaborate with researchers running human studies to translate human evaluation frameworks into automated or semi-automated systems.
- Partner with model researchers to identify failure cases and build targeted evaluation harnesses.
- Maintain dashboards, reporting tools, and alerting systems to surface evaluation results to stakeholders.
- Stay current with emerging evaluation techniques in generative AI, multimodal LLMs, and perceptual quality assessment.
Qualifications
- Master's or PhD in Computer Science, Machine Learning, or a related technical field (or equivalent industry experience).
- 5+ years of experience building ML evaluation systems, model pipelines, or large-scale infrastructure.
- Hands-on experience working with visual data (images and/or video), including evaluation, modeling, or data preparation.
- Proficiency in Python and ML frameworks (PyTorch, JAX, or TensorFlow).
- Familiarity with human-in-the-loop evaluation workflows and how to scale them with automation.
- Strong background in machine learning, with experience in generative models (diffusion, LLMs, multimodal architectures).
- Strong software engineering skills (CI/CD, testing, data pipelines, distributed systems).
Skills
Required:ML evaluation system designPythonML frameworksVisual data handling
Nice to have:Human-in-the-loop evaluation
Last updated Jul 22, 2026. Interested? Apply on the company site.
About Luma AI
Create, animate & innovate with Luma’s AI.
Create, animate & innovate with Luma’s AI. Use text, images, or video to generate realistic motion content with Ray3 and Dream Machine for next-gen storytelling.
AI Video GenerationAPIDream MachineMediaRay3liquid-ai-recruiting-target
More open roles at Luma AI
- Applied Research Scientist / Engineer3mo agoHybrid (New York, NY; SF Bay Area, CA; London, UK) · Applied Research & Engineering
- Research Scientist / Engineer – Reinforcement Learning Infrastructure3mo agoHybrid (SF Bay Area, CA), Remote +1 more · Systems Research & Engineering
- Product Manager, Applied Research3mo agoHybrid (SF Bay Area, CA) · Product Management
- Forward Deployed Engineer3mo agoHybrid (SF Bay Area, CA) · Product Engineering
- Product Manager, Creative Tools3mo agoHybrid (SF Bay Area, CA) · Product Management