Data Scientist - Evaluations, Chanakya
BengaluruFull-timePosted 3mo ago
About the role
Data Scientist anchoring evaluations for Sarvam’s vertical, designing and maintaining evaluation frameworks to measure model and system quality in operational context, focused on domain-specific, high-stakes use cases.
Responsibilities
- Design and build evaluation frameworks for Sarvam's AI outputs across domain-specific requirements: document comprehension, command summarisation, geospatial reasoning, enterprise workflow automation, and others as they emerge
- Define quality metrics in collaboration with domain experts and clients; translate operational requirements into measurable, defensible signals
- Run structured evaluation cycles pre- and post-deployment; build dashboards that surface model quality in production
- Identify failure modes, edge cases, and distribution shifts — with a bias for finding what's wrong, not what's right
- Collaborate with the MLOps Engineer to operationalise eval pipelines — automated, triggered by deployment events, versioned, and reproducible
- Build and manage domain-specific datasets for fine-tuning, evaluation, and benchmarking — including human annotation workflows where needed
- Publish internal findings and quality reports that feed the product and engineering roadmap
Qualifications
- 3–6 years in data science, ML research, or applied AI; at least 2 years working with LLMs in production contexts
- Strong statistics and probability fundamentals — understanding of valid vs misleading evaluations
- Experience designing evaluation frameworks from scratch: custom metrics, inter-rater reliability, red-teaming methodologies
- Python proficiency; comfort with pandas, NumPy, HuggingFace datasets, RAGAS, EleutherAI Eval Harness, LangSmith, or equivalent
- Experience with prompt engineering, model fine-tuning, or RLHF in applied settings
- Ability to work with unstructured domain data: PDFs, doctrine documents, transcripts, and field reports
Skills
Required:Evaluation designPython
Benefits
- Note: Benefits not explicitly listed in the description; not included as filler values.
Last updated Jul 21, 2026. Interested? Apply on the company site.
About Sarvam
Sarvam | India's Full-Stack Sovereign AI Platform
Sarvam is India's full-stack sovereign AI platform, with speech-to-text, text-to-speech, translation, and conversational agents across 22 Indian languages. Start free.
API PricingAbout UsAksharAryaCareersConversationsDocument DigitisationDocumentation
More open roles at Sarvam
- Software Engineer - Model API Platform1w agoBengaluru · Engineering
- Staff Engineer - Systems Engineering2w agoBengaluru · Engineering
- Staff Engineer - Telephony2w agoBengaluru · Engineering
- Growth Marketing Intern3w agoBengaluru · Marketing
- Marketing Intern4w agoBengaluru · Marketing