Senior Technical Product Manager, ML Quality

Summary

Join WellSaid, a leading AI voice company, as a Senior Technical Product Manager, ML Quality. You will manage the development of products and processes for evaluating data, models, and features developed by the ML team. Responsibilities include defining QA strategy for ML models, evaluating and validating TTS models, managing data quality, developing QA automation frameworks, monitoring model performance, and moderating ethical considerations. You will collaborate closely with data engineers and software engineers. The ideal candidate possesses 2+ years of Machine Learning Product Management experience, 5+ years in ML, TTS, or related fields, and strong technical skills. WellSaid offers competitive salary and stock options, full medical, dental, and vision insurance, a matching 401(k) plan, generous paid time off, parental leave, and a learning & development stipend.

Requirements

• 2+ Machine Learning Product Management experience at a similar-stage company
• 5+ years in ML, TTS, or related fields with strong instincts for identifying market differentiators
• Tenured understanding and experience with ML research, testing, and deployment
• Strong cross-team collaboration skills, especially collaborating with ML and Data Engineers
• Experience managing multiple projects at once with a track record of predictable, on-time, on-budget delivery
• Clear, direct communication skills (written and verbal)
• Organization skills that demonstrate you are process-oriented
• Strong ability to create order from ambiguity
• Kind, empathetic, and intentional leadership qualities
• Must be a U.S. Citizen or Permanent Resident
• Must pass a pre-employment background check

Responsibilities

• Define QA strategy for ML models across many languages: Identifying critical metrics and testing methodologies to assess the performance of TTS models throughout the research & development lifecycle. This includes developing such strategies for any new languages we explore in the future
• Evaluate and validate TTS models: Working with the Applied ML team and selecting appropriate evaluation metrics, running rigorous tests across datasets, experiments, and final models, and interpreting results to identify areas for improvement and/or offer recommendations to the Research team
• Manage data quality: Monitoring data quality, identifying potential biases in training data, and ensuring data preprocessing is done correctly to optimize model performance
• Develop QA automation frameworks: Defining automated testing tools and pipelines to streamline the testing and evaluation processes for TTS models, especially for regression testing
• Monitor model performance in production: Tracking key metrics post-deployment to identify potential degradation in model performance and take corrective action. This may also require defining the monitoring and tooling needed to do so
• Moderate ethical considerations: Assessing potential biases in ML models and training data, ensuring responsible development practices, and addressing concerns regarding fairness, transparency, and Responsible AI
• Research and partner with third parties: Defining test cases, scenarios, and evaluation batches with external evaluation services when appropriate. Recommending evaluation services to leadership with a well-defined testing strategy defending your recommendation
• Collaborate closely with data engineers and software engineers in the ML team to understand model capabilities, identify new methods or metrics for gauging model progression, and create new tools to make the testing and evaluation processes fast and easy to implement
• Write design docs with clear requirements and expectations for engineers to execute against
• Research and collate text and audio test cases from real customer data, product feedback, and active experiments
• Synthesize evaluation results into reports for internal and external publication and/or deliver recommendations to the research team informed by rigorous regression analysis
• Partner closely with Lead Product Manager, ML on overall product roadmap, customer impact, and development recommendations

Preferred Qualifications

• Previous ML engineering experience
• Experience in audio signal processing or linguistic support at the machine learning level

Benefits

• Competitive salary and stock options
• Full medical, dental, and vision insurance
• Matching 401(k) plan
• Generous vacation policy/paid time off
• Parental leave
• Learning & development stipend
• Home office stipend