Remote AI Evaluator

Logo of Sama

Sama

๐Ÿ“Remote - India

Job highlights

Summary

Join us as an AI Evaluator (English and Hindi) to help state-of-the-art generative AI models perform better. You will analyze model responses, identify policy concerns, and ensure quality within a defined SLA.

Requirements

  • Have completed a Bachelor's degree
  • Fluency in Hindi at a level equivalent to C2 level
  • Fluency in English at C1 level
  • Digital skills for high levels of research to find accurate information from reliable internet sources
  • Strong critical thinking, reasoning, and exceptional problem-solving skills
  • Excellent attention to detail

Responsibilities

  • Analyze and understand the meaning of the model response, the user feedback and how it relates to the response
  • Identify messages that are inappropriate and trigger policy concerns
  • Define the target information (claims) in the response
  • Recognize information that isnโ€™t clearly pointed out as wrong - e.g. missing details that make the response incomplete - or information thatโ€™s mixed up between multiple entities (disambiguation)
  • Formulate appropriate strings for fact-checking and web research that helps them find the right sources quickly
  • Recognize any differences between the text in the input passage and the text generated in the response, thus identifying input issues
  • Understand the context of the user-model interaction, and recognize when the model doesnโ€™t interpret that context correctly
  • Workflow-related: understand complex Guidelines instructions, ask well-formulated questions, leave informative justifications and task comments
  • Understand the client platform and follow team level protocols (like how/when to access the task, etc.)

Preferred Qualifications

  • Excellent written and oral communication skills in both English and Hindi
  • Hindi qualifications are preferred at Hindi Honors or MPhil levels

Benefits

  • Competitive compensation corresponding to market data & level of experience
  • Medical, dental, vision, and life insurance
  • Holiday and vacation policies
  • Professional development stipends
  • Paid sabbatical program

Share this job:

Disclaimer: Please check that the job is real before you apply. Applying might take you to another website that we don't own. Please be aware that any actions taken during the application process are solely your responsibility, and we bear no responsibility for any outcomes.

Similar Remote Jobs

Please let Sama know you found this job on JobsCollider. Thanks! ๐Ÿ™