Data Operations, Ai Evaluations

Summary

Join The Browser Company and help build the foundation for Dia, our browser-native AI assistant, as a Data Operations, AI Evals specialist. You will be responsible for creating high-quality datasets for model evaluation and training. This involves collaborating with engineers, product owners, and user research teams to understand user needs and translate them into effective training data. You will use various tools and techniques to manage and update training data, ensuring quality standards are met. Your work will directly impact Dia's success by enabling our AI to understand user intent and deliver helpful responses. The role requires significant experience in AI evaluation, data labeling, and working with large datasets. The Browser Company offers a competitive salary and benefits package, including comprehensive health insurance, 401k, flexible vacation, remote work options, and paid parental leave.

Requirements

• Have 3+ years of hands-on experience with AI evaluation (”evals”), data labeling, or model fine-tuning. Have a strong understanding of best practices in these areas
• Have 5+ years of experience working with large datasets, from spreadsheets to user feedback, in a technical, product, or QA role
• Understand how users think about product capabilities, can distinguish between current features and future potential, and collaborate across teams to turn insights into action
• Be comfortable with technical tools like GitHub and can navigate engineering-adjacent systems
• Be excited about AI, language models, and taking creative approaches to dataset creation to ensure diverse, high-quality examples for AI training and evaluation
• Have 4+ hours of overlap time with team members in Eastern Time Zone

Responsibilities

• Build high-quality datasets for model evaluation and training, from targeted eval sets to large-scale training data
• Partner with engineers to ensure datasets are comprehensive, properly formatted, and easy to use
• Work with product owners to understand product goals and translate them into effective training data
• Collaborate with User Research and Membership teams to understand user needs deeply
• Use support tickets and user feedback to inform and inspire dataset creation
• Establish and maintain quality standards for our datasets
• Navigate technical tools to manage and update training data

Preferred Qualifications

Have Sqlite, Python, Braintrust, or Xcode experience

Benefits

• Flexible compensation model with options for salary-optimized, equity-optimized, and balanced offers
• Annual salary range of $140,000- $190,000 USD
• Comprehensive benefits package with employee medical, dental, and vision - 100% of premiums covered for employees, and up to 95% for dependents
• 401k plan
• Flexible vacation policy - on average, our team members take between 15-20 vacation days plus federal holidays
• Remote-friendly working environment - core working hours are 11 AM-2 PM Eastern Time, Monday-Friday
• 12 weeks of paid parental leave
• $1,500 USD home office stipend
• Free annual memberships to One Medical (where available), Talkspace, Teladoc, and HealthAdvocate (for US-based employees)