Senior Data Engineer

Summary

Veterinary Practice Partners is seeking a Senior Data Engineer to join their team full-time. The role involves building exceptional data capabilities for partner hospitals, developing and maintaining high-quality data pipelines, optimizing performance, data governance, technical leadership, strategy creation, documentation, third-party management, and more. The salary range starts at $120,000 based on experience and skill set.

Requirements

• 3+ years in data engineering roles in a production environment
• Advanced proficiency in Python and SQL for data engineering
• Up-to-date knowledge of and 1+ years of experience using Databricks for Lakehouse management
• Deep understanding of data modeling, data architecture, and data integration best practices
• Strong hands-on experience with Apache Spark
• Familiarity with data governance, security, and privacy principles
• Comfort using git or equivalent to manage the software development life cycle
• Exceptional ability to learn and use new software development techniques and tools
• Ability to manage multiple projects simultaneously
• High energy, humble team player with “get it done” attitude, seeking collaboration with colleagues

Responsibilities

• Define and execute processes needed to develop, test, deploy, and maintain high quality data pipelines
• Oversee the end-to-end development of data pipelines from source data extraction through to production-grade analytical dataset delivery, ensuring data quality and security throughout the pipeline
• Continuously monitor and optimize data processing performance and efficiency. Identify and address bottlenecks, optimize query performance, and improve overall system stability
• Establish and enforce data quality management policies, data access controls, and data privacy standards
• Stay abreast of the latest developments in engineering tools and best practices. Provide guidance to the team about technical challenges
• Collaborate with cross-functional teams to define the data engineering strategy aligned to business objectives, including data modeling that unifies data assets across a range of source systems used to manage the operations of our partnering hospitals
• Maintain clear and comprehensive documentation of data pipelines, architecture, and processes to ensure knowledge sharing and team continuity
• Evaluate and manage relationships with third-party vendors and tools, making informed decisions about when to leverage external solutions

Preferred Qualifications

• Experience with the Azure cloud ecosystem
• Experience developing production-ready, real-time machine learning model serving pipelines
• Comfort developing in the Apache Spark Structured Streaming paradigm
• Experience working in a private equity-backed services company
• Experience deploying machine learning models with MLFlow or equivalent
• Experience developing CI/CD pipelines