Remote AI Evaluator

Summary

Join us as an AI Evaluator (English and Hindi) to help state-of-the-art generative AI models perform better. You will evaluate LLM responses for accuracy and policy sustainability, working in a fully remote position.

Requirements

• Have completed a Bachelor's degree
• Fluency in Hindi at a level equivalent to C2 level
• Fluency in English at C1 level
• Digital skills for high levels of research to find accurate information from reliable internet sources
• Strong critical thinking, reasoning, and exceptional problem-solving skills
• Excellent attention to detail

Responsibilities

• Analyze and understand the meaning of the model response, the user feedback and how it relates to the response
• Identify messages that are inappropriate and trigger policy concerns
• Define the target information (claims) in the response
• Recognize information that isn’t clearly pointed out as wrong - e.g. missing details that make the response incomplete - or information that’s mixed up between multiple entities (disambiguation)
• Formulate appropriate strings for fact-checking and web research that helps them find the right sources quickly
• Recognize any differences between the text in the input passage and the text generated in the response, thus identifying input issues
• Understand the context of the user-model interaction, and recognize when the model doesn’t interpret that context correctly
• Workflow-related: understand complex Guidelines instructions, ask well-formulated questions, leave informative justifications and task comments
• Understand the client platform and follow team level protocols (like how/when to access the task, etc.)

Preferred Qualifications

• Excellent written and oral communication skills in both English and Hindi
• Hindi qualifications are preferred at Hindi Honors or MPhil levels

Benefits

• Competitive compensation corresponding to market data & level of experience
• Medical, dental, vision, and life insurance
• Holiday and vacation policies
• Professional development stipends
• Paid sabbatical program