← All roles

Data Annotation Specialist, Generalist

Cohere· Data Quality (Contract)
Apply on company site
RemoteCA$30kContractPosted 2mo ago

About the role

Part-time independent contractor role in Canada for performing data annotation and model evaluation across text, image, and structured data to improve large language models. 16 hours/week at CAD 30/hour, remote, BYOD. Ambitious, judgment-driven work with evolving tasks and strict emphasis on data quality and consistency.

Responsibilities

  • Evaluate and rank model outputs: Complete preference and comparison tasks, assessing which responses best conform to project guidelines for accuracy, helpfulness, tone, and safety, and writing clear justifications for your judgments.
  • Stress-test and break models: Probe models adversarially to surface failure modes, unsafe behavior, and capability gaps, and document reproducible cases that engineering and research teams can act on.
  • Create datasets: Author high-quality prompts, responses, and exemplars to build training and evaluation datasets, following detailed specifications and editing machine-written or human-written outputs to standard.
  • Build and apply rubrics and taxonomies: Contribute to the design of grading criteria and rubrics, then apply them consistently to produce structured, high-quality annotations across task types.
  • Annotate and correct multimodal data: Label, audit, and rectify inaccuracies across text, image, and structured data, maintaining a high standard of data integrity and accuracy.
  • Calibrate and maintain consistency: Participate in calibration exercises and inter-annotator agreement checks to align on standards, and flag ambiguous or uncovered edge cases rather than guessing.
  • Adapt to experimental work: Take on new and evolving task types as project needs shift, applying sound judgment in areas where guidelines are still being developed.
  • Report on model performance: Surface and communicate quality and performance trends in model and agent behavior, giving cross-functional partners clear, well-evidenced feedback.

Qualifications

  • 1+ years of experience in AI data annotation, LLM evaluation, content moderation, research, or a related analytical role, with exposure to quality assurance and/or preference ranking.
  • Experience applying detailed guidelines to complex and often ambiguous content with strong contextual and sociocultural judgment, sensitivity to nuance, tone, and register, and ability to reason when there is no single correct answer.
  • Comfort with ambiguity and willingness to flag unclear or uncovered edge cases rather than guess, and to work on novel, experimental tasks with evolving definitions.
  • Sharp eye for inconsistencies, subtle errors, and model failure modes; familiarity with LLM behaviors like hallucination and instruction-following gaps is a plus.
  • Excellent command of written English and strong reading comprehension; ability to justify evaluations and write clear prompts and exemplars; multilingual ability is a bonus.
  • Strong attention to detail and ability to maintain consistency on high-volume tasks.
  • Comfort with annotation platforms and structured formats such as JSON, CSV/TSV, Markdown, XML, and YAML.
  • Strong remote-work execution, time management, and ability to work independently in a global, asynchronous team.

Skills

Required:AI data annotationLLM evaluationQuality assurance
Nice to have:Content moderation
Last updated Oct 21, 2026. Interested? Apply on the company site.
Cohere logo

About Cohere

Support automation that's actually intelligent

Cohere resolves 40-70% of support requests using AI, over all channels, within just a few days.

artificial-intelligenceautomationliquid-ai-recruiting-target
Company profileWebsite

More open roles at Cohere