Software Engineer in Test

Summary

Join a forward-thinking engineering team as a Software Engineer in Test (AI & Automation Focus) to build and evaluate AI-powered systems, including a sophisticated voice assistant platform. This unique role blends test automation, machine learning evaluation, and data analysis. You will design test automation frameworks, evaluate large language model (LLM) performance, and develop custom tooling to analyze and monitor product quality. Responsibilities include building automated evaluation pipelines for LLMs, analyzing large-scale data to identify issues, creating custom scripts for automation and log parsing, and developing visual dashboards. You will also support test environment setup, perform root cause analysis, and drive innovation by introducing new testing methodologies. This position requires collaboration with cross-functional teams.

Requirements

• BS degree in Computer Science or related technical field
• 3+ years of experience in test automation , ML evaluation , data analysis , or related roles
• Proficient in JavaScript/TypeScript and Python ; strong scripting experience, especially for automation and log parsing
• Experience with tools such as Jest , Playwright , Cypress , or Selenium for UI and API automation
• Strong knowledge of SQE principles , risk-based testing, test planning, and system integration
• Knowledge of testing approaches for AI based products, such as AI evaluations metrics and their usage
• Familiarity with LLMs , GenAI , and prompt engineering
• Comfortable working with API testing , log data analysis , and performance metrics
• Experience in evaluating and testing non-deterministic systems AI-powered projects such as conversational models or recommendation engines
• Solid understanding of CI/CD pipelines , cloud-based environments , and microservices/serverless architectures
• Strong communication and analytical skills; ability to clearly convey technical findings to both technical and non-technical audiences
• Self-starter with a growth mindset and a passion for building scalable, intelligent testing systems

Responsibilities

• Design, build, and maintain robust, data-driven test automation frameworks JavaScript/TypeScript and Python
• Build automated evaluation pipelines and prompt-based tests for LLM and conversational AI systems
• Analyze large-scale, unstructured logs and telemetry data to identify issues and user experience gaps
• Create custom scripts to support test automation, log parsing, and performance evaluation of machine learning models
• Develop visual dashboards and reports to track quality, system health, and model behavior over time
• Support test environment setup and integrate with CI/CD pipelines for continuous testing
• Perform root cause analysis , debugging, and reporting of product issues, collaborating closely with Developers and SQEs across the SDLC
• Drive innovation by introducing new testing methodologies , tools, and best practices for both deterministic and non-deterministic systems
• Partner with global, cross-functional teams including ML engineers, product managers, and infrastructure leads

Benefits

• Competitive Benefits - Employee benefits include comprehensive medical, dental, vision, life and disability plans for eligible employees and their dependents
• Additionally, Natera employees and their immediate families receive free testing in addition to fertility care benefits
• Other benefits include pregnancy and baby bonding leave, 401k benefits, commuter benefits and much more
• We also offer a generous employee referral program!