Principal Software Engineer

Summary

Join ExtraHop as a Principal Software Engineer | MLI and contribute to the design and implementation of scalable systems supporting machine learning projects. Collaborate with data scientists, ML engineers, and cloud teams to build efficient tools and infrastructure for managing machine learning workflows. Lead engineering best practices through code reviews, mentorship, and high standards for security, performance, and reliability. Improve machine learning processes by focusing on clear documentation, easy version management, automated deployments (CI/CD), and secure operations. Stay informed on new developments in MLOps, container management, and cloud technologies to guide decisions for a modern and effective ML platform. Help shape the future of cybersecurity by working on a market-leading NDR product.

Requirements

• 8+ years of professional software engineering experience, with 3+ years focused on ML infrastructure, distributed systems, or platform engineering
• Strong programming skills in Python and Go, with an emphasis on writing production-quality, well-tested, and modular code
• Demonstrated experience building and scaling systems that support data-intensive or ML workloads in cloud or hybrid environments
• Solid experience with IaC tools such as Terraform, and configuration management systems (e.g., Ansible, Packer)
• Familiarity with containerization and orchestration technologies (e.g., Docker, Kubernetes), especially in support of ML workloads
• Working knowledge of ML model lifecycle, MLOps best practices, and ML frameworks (e.g., PyTorch, TensorFlow, MLflow)
• Experience with CI/CD pipelines, automation frameworks, and observability tools (e.g., Prometheus, Grafana, Datadog)
• Excellent problem-solving skills with the ability to lead technically complex projects from concept to production
• U.S. citizenship or lawful permanent resident status required for work in secure environments

Responsibilities

• Provide technical leadership in the architecture, design, and implementation of robust and scalable infrastructure to support the full lifecycle of machine learning systems—from data ingestion to model deployment and monitoring
• Collaborate with data scientists, ML engineers, and cloud teams to build high-performance services, pipelines, and tooling that accelerate ML experimentation and production workflows
• Drive the evolution of ML infrastructure capabilities, including support for reproducibility, versioning, CI/CD for ML models, observability, and secure deployment
• Design and develop infrastructure-as-code (IaC) solutions using Terraform or similar tools to ensure infrastructure scalability, consistency, and automation
• Champion engineering excellence through code reviews, mentorship, and by setting high standards for performance, security, and maintainability
• Stay informed about industry trends in ML infrastructure, MLOps, container orchestration, and cloud-native tools, and help shape strategic technology decisions
• Contribute to internal documentation, operational runbooks, and design specifications to enable knowledge sharing and operational resilience across the team

Preferred Qualifications

• Master’s or Bachelor's degree in Computer Science, Engineering, or a related field
• Prior experience securing ML systems or data pipelines, with an understanding of role-based access control (RBAC), service accounts, and network segmentation
• Familiarity with compliance and regulatory standards such as FedRAMP, NIST SP 800-53, or similar
• Exposure to security in ML and cloud environments, including threat modeling, guardrails, and vulnerability scanning tools
• AWS, GCP, or Azure certification (e.g., Solutions Architect or ML Specialist) is a plus
• Experience supporting or building Network Detection and Response (NDR), intrusion detection systems, or other cybersecurity-focused ML applications is highly desirable

Benefits

• Health, Dental, and Vision Benefits
• Flexible PTO, Sick Time Prorated Based on Date of Hire, and All Federal Holidays (US Only) + 3 Days of Paid Volunteer Time
• Non-Commissioned Positions may be eligible to participate in the Annual Discretionary Bonus Plan
• FSA and Dependent Care Accounts + EAP, where applicable
• Educational Reimbursement
• 401k with Employer Match or Pension where applicable
• Pet Insurance (US Only)
• Parental Leave (US Only)
• Hybrid and Remote Work Model