Remote AI/ML Engineer

Summary

The Senior Data Engineer will be responsible for designing, implementing, and maintaining scalable data processing solutions and machine learning pipelines using GCP services like Vertex AI, BigQuery ML, and AI Platform Pipelines. Key responsibilities include integrating AI services, managing model lifecycle, optimizing query performance in BigQuery, and ensuring data security.

Requirements

• Extensive experience with GCP services, including Vertex AI, BigQuery ML, and AI Platform Pipelines
• Strong understanding of data lake architectures and scalable data processing solutions
• Familiarity with data versioning, metadata management, and data quality assurance practices
• Excellent problem-solving skills and ability to handle complex data engineering challenges

Responsibilities

• Integrate Vertex AI with BigQuery for training machine learning models
• Deploy machine learning models using Vertex AI, including exporting, versioning, and monitoring
• Automate ML workflows using Vertex AI Pipelines
• Implement data versioning and tracking using Vertex AI Dataset service and metadata stores
• Set up monitoring and logging for AI models deployed on Vertex AI using Google Cloud Monitoring and Logging
• Integrate Gemini with Vertex AI for enhanced model training and predictions
• Use Gemini for feature engineering in Vertex AI workflows
• Design scalable data lake architectures using BigQuery
• Optimize query performance in a BigQuery-based data lake through partitioning, clustering, and materialized views
• Ingest data from various sources (e.g., Kafka, Pub/Sub) into BigQuery
• Handle data security and access control in a BigQuery data lake

Preferred Qualifications

• Advanced degree in Computer Science, Data Engineering, or a related field
• Certifications in Google Cloud Platform
• Knowledge of Kubeflow Pipelines and CI/CD practices for ML workflows