Senior DataBricks Engineer

Summary

Join our team as a Senior DataBricks Engineer with Machine Learning experience for a fully remote, full-time position with a US-based client! This 3-month project (with potential for extension) involves migrating an ML model from AWS to DataBricks. You will be responsible for tasks such as assessing project requirements, preparing the platform for integration, data processing and transformation, implementing MLOps with MLflow, and creating testing frameworks. The client values effective communication and teamwork. Flexible working hours are available.

Requirements

• 6+ years of experience in data engineering/data science
• Very good knowledge of Apache Spark and the DataBricks platform
• Solid experience in ML areas
• Experience with MLOps and MLflow
• Experience working in the AWS environment
• Experience in conducting similar migrations
• Interpersonal and teamwork skills – we value people who emphasize effective (not necessarily efficient 😄) communication
• Ability to take initiative and independence
• English at a level that allows for fluent communication in a team

Responsibilities

• Assess project requirements, analyze the current architecture, and create a new model architecture
• Prepare the platform for integration with Databricks and ensure Unity Catalog compatibility and configuration
• Process and transform data with metric aggregation for two related Glue tasks
• Create ETL pipelines to process data on platform visits and users, including data flattening for model needs
• Implement MLOps using MLflow
• Run the model on Databricks Serving Endpoints to test latency
• Create testing frameworks and support the Tealium team in testing
• Prepare an implementation plan for live launch
• Document work results using Unity Catalog

Preferred Qualifications

• Experience in designing and optimizing data flows using DBT, SSIS, TimeXtender, or similar solutions (ETL)
• Experience with any big data or NoSQL platforms (Redshift, Hadoop, EMR, Google Data, etc.)

Benefits

• Private medical care (Medicover)
• Multisport card for contractors