Gen Ai Tester

Summary

Join our team as a Generative AI Model Tester with 7+ years of experience. You will be responsible for creating and executing test strategies and plans for various AI models, including LLMs. Your expertise in NLP, AI/ML concepts, and testing frameworks is crucial. You will perform functional, load, and stress tests, validate model output, and ensure data accuracy. Strong communication and collaboration skills are essential for working with cross-functional teams. Proficiency in Python or Java for automated testing is required.

Requirements

• Have hands-on experience in testing Generative AI models (text, image, other content)
• Have 7+ years of overall experience
• Possess a strong understanding of AI/ML concepts, including model training, validation, deployment, and continuous monitoring
• Be proficient in testing large language models (LLMs) such as GPT, BERT, and similar
• Have expert-level knowledge in natural language processing (NLP) techniques
• Have experience with AI/ML testing frameworks and tools such as TensorFlow, PyTorch, Hugging Face or custom AI testing frameworks
• Be familiar with data validation and testing
• Be proficient in defining KPIs and metrics for generative AI model testing
• Have understanding of cloud-based AI/ML deployment and experience in testing AI models deployed on cloud platforms
• Be proficient in API testing for AI/ML applications
• Have experience in using test automation tools for AI/ML testing
• Be proficient in programming languages like Python or Java for developing and executing automated test scripts for AI models
• Have knowledge of Continuous Integration/Continuous Deployment (CI/CD) pipelines in the context of AI/ML model deployment and testing
• Possess strong stakeholder management and communication skills
• Possess strong analytical, problem-solving, and reporting skills
• Have excellent communication skills

Responsibilities

• Create and execute test strategies, test plans, and cases specific to generative AI models
• Test generative AI models, including language models, image generators, and other AI applications
• Test for potential biases within AI models by analyzing model output across different demographics and data segments
• Define KPIs and metrics for generative AI model testing
• Conduct model evaluation using relevant performance metrics, such as BLEU, ROUGE, and perplexity for language models
• Validate model output for accuracy, coherence, and relevance
• Optimize queries and process data to validate AI model performance and output efficiency
• Perform functional, load, and stress tests on models
• Test AI models deployed on cloud platforms like AWS, Azure, or GCP
• Perform API testing for AI/ML applications
• Use test automation tools for AI/ML testing, including custom automation scripts
• Develop and execute automated test scripts for AI models using programming languages like Python or Java
• Work with CI/CD pipelines in the context of AI/ML model deployment and testing
• Perform continuous testing for model performance and drift post-deployment
• Collaborate with cross-functional teams, including data scientists, developers, ML Engineers, Model Ops team and product managers
• Provide feedback on model performance and potential improvements
• Use Model Ops tools and practices for production-level AI testing
• Identify and resolve issues related to AI model performance and integration
• Present test results and model evaluations to technical and non-technical stakeholders
• Use test management tools like JIRA, TestRail, or similar for end-to-end test management

Preferred Qualifications

• Have familiarity with Model Ops tools and practices for production-level AI testing
• Have exposure and experience in test management tools like JIRA, TestRail, or similar