Ai Evaluator

Summary

Join Sama as an AI Evaluator and contribute to improving state-of-the-art generative AI models. You will analyze model responses in both English and Hindi, ensuring accuracy and policy compliance. This fully remote position requires proficiency in both languages and strong critical thinking skills. You will evaluate LLM responses, identify inappropriate content, and formulate fact-checking queries. The role involves understanding complex guidelines, following protocols, and collaborating with delivery managers. A Bachelor's degree and fluency in Hindi and English are required.

Requirements

• Have completed a Bachelor's degree
• Fluency in Hindi at a level equivalent to C2 level
• Fluency in English at C1 level
• Digital skills for high levels of research to find accurate information from reliable internet sources
• Strong critical thinking, reasoning, and exceptional problem-solving skills
• Excellent attention to detail

Responsibilities

• Analyze and understand the meaning of the model response, the user feedback and how it relates to the response
• Identify messages that are inappropriate and trigger policy concerns
• Define the target information (claims) in the response
• Recognize information that isn’t clearly pointed out as wrong - e.g. missing details that make the response incomplete - or information that’s mixed up between multiple entities (disambiguation)
• Formulate appropriate strings for fact-checking and web research that helps them find the right sources quickly
• Recognize any differences between the text in the input passage and the text generated in the response, thus identifying input issues
• Understand the context of the user-model interaction, and recognize when the model doesn’t interpret that context correctly
• Workflow-related: understand complex Guidelines instructions, ask well-formulated questions, leave informative justifications and task comments
• Understand the client platform and follow team level protocols (like how/when to access the task, etc.)
• Strong logic, intuitive problem-solving and critical thinking; adaptability to the needs of the task. The candidate needs to be open-minded and welcome different viewpoints or solutions
• Proactiveness, confidence to work independently without constant support. Reliability and being able to make good decisions on their own
• Be a thought partner to delivery managers, input into the project delivery strategies, and share quality best practices to aid in achieving the project goals

Preferred Qualifications

• Excellent written and oral communication skills in both English and Hindi
• Hindi qualifications are preferred at Hindi Honors or MPhil levels
• Demonstrable ability to perform well in a rapidly changing and extremely global team
• Passion for our mission of ensuring a world-class support experience for our community and customers
• Experience as an editor is an added advantage

Benefits

• Competitive compensation corresponding to market data & level of experience
• Benefits packages based on country of employment, including but not limited to medical, dental, vision, and life insurance
• Holiday and vacation policies
• Professional development stipends
• A paid sabbatical program