Senior Data Engineer

Summary

Join Motional as a Senior Data Engineer and play a pivotal role in enhancing the performance of our autonomous vehicle (AV) technology. You will collaborate with cross-functional teams to design, develop, and implement key performance metrics for AV systems. Leverage your expertise in Python to build robust data pipelines for processing vast amounts of AV data. Design and implement data models and schemas optimized for storing and querying AV performance metrics. Ensure data quality and accuracy through rigorous validation processes. Optimize data infrastructure for performance and scalability on cloud platforms like AWS or GCP. Automate data pipeline deployments and infrastructure provisioning using IaC principles. Document your work and share best practices with the team. Continuously learn and adapt to new technologies in the AV and data engineering fields. This role offers a unique opportunity to make a significant impact on the future of autonomous vehicles.

Requirements

• Minimum of 5-7 years of experience in data engineering roles
• Bachelor's degree in Computer Science, Engineering, Robotics, or a related field. A Master's degree is a plus
• Expert-level proficiency in Python and strong understanding of software development principles
• Hands-on experience with one or more data pipeline orchestration tools
• Experience with distributed data processing frameworks
• Strong understanding of relational databases and data warehousing concepts
• Experience working with cloud data services on AWS, Azure, or GCP
• Proven ability to design and implement effective data models for complex data
• Deep understanding of ETL and ELT concepts and best practices
• Proficient with Git and related version control workflows
• Strong analytical and problem-solving skills, particularly in the context of complex system performance evaluation
• Excellent communication and collaboration skills, with the ability to effectively communicate technical concepts to diverse audiences
• Ability to work independently and as part of a multidisciplinary team
• Strong attention to detail and a commitment to data quality and accuracy
• Proactive and self-motivated with a passion for autonomous vehicle technology
• Comfortable working in a fast-paced, continuous delivery environment

Responsibilities

• Design and Development of AV Performance Metrics: Collaborate with cross-functional teams, including systems engineers, software engineers, AI scientists, control engineers, simulation engineers, and validation teams to understand their needs and define, design, and implement key metrics to evaluate various aspects of autonomous vehicle performance (e.g., lane change capability performance, oscillation and braking metrics)
• Software Development for Metrics and Analysis: Leverage strong Python programming skills to develop framework, libraries and tools for calculating, aggregating, and visualizing AV performance metrics from large datasets
• Building Scalable Data Pipelines for AV Data: Design, develop, and maintain efficient and scalable data pipelines to ingest, process, transform, and store vast amounts of data generated by autonomous vehicles (e.g., sensor data, simulation data, logs, annotations)
• Data Modeling and Storage for AV Metrics: Design and implement data models and schemas optimized for storing and querying AV performance metrics, considering factors like granularity, temporal aspects, and dimensionality
• Data Quality and Validation for AV Data: Implement rigorous data quality checks and validation processes to ensure the accuracy, reliability, and integrity of the data used for performance evaluation
• Performance Optimization of AV Data Infrastructure: Identify and implement performance optimizations for data pipelines and storage solutions to handle the high volume and velocity of AV data
• Cloud Infrastructure for AV Data: Design, build, and manage data infrastructure on cloud platforms (e.g., AWS, GCP) specifically tailored for AV data processing and storage
• Automation and Infrastructure as Code: Implement automation for data pipeline deployments, infrastructure provisioning, and metric calculation processes using Infrastructure as Code (IaC) principles and tools
• Documentation and Knowledge Sharing: Create clear and concise documentation for data pipelines, metrics definitions, and analysis tools. Share knowledge and best practices with the wider engineering team
• Staying Current with AV and Data Engineering Technologies: Continuously learn and evaluate new technologies and methodologies relevant to autonomous vehicles and data engineering

Preferred Qualifications

• Experience working with autonomous vehicle sensor data (e.g., LiDAR, camera, radar)
• Familiarity with simulation environments and data generated from AV simulations
• Experience with data visualization tools (e.g., Looker) for presenting performance metrics
• Knowledge of statistical analysis and machine learning techniques relevant to performance evaluation
• Experience with real-time data processing and streaming technologies
• Familiarity with data annotation pipelines and tools
• Familiarity with Ray and Ray clusters for scaling Python applications and AI/ML tasks

Benefits

Medical, dental, vision, 401k with a company match, health saving accounts, life insurance, pet insurance