Databricks Architect

Summary

Join CoEnterprise's growing analytics practice as a US-based remote Databricks Architect. You will design and implement innovative data solutions using Databricks and Medallion Architecture, addressing complex business challenges for clients. This long-term role involves leading cutting-edge data initiatives and contributing to the practice's strategic growth. You will collaborate with clients and internal stakeholders, delivering robust, scalable, and automated data architectures. The position requires a blend of technical expertise, hands-on development, and collaboration skills. This is a key leadership role within a supportive and high-energy team.

Requirements

• 7+ years’ experience in data engineering or software development, with a strong focus on data architecture
• Extensive experience (3-5+ years) with Databricks and the Medallion Architecture
• Expertise in Python, including developing production-grade data pipelines, reusable automation scripts, and leveraging libraries such as Pandas, and NumPy for advanced data manipulation and analytics
• Proficiency in PySpark and SQL with in-depth experience integrating a wide variety of data sources with Databricks
• Expertise in cloud platform services (AWS, Azure, GCP), including their data storage solutions (e.g., S3, Azure Data Lake Storage, Google Cloud Storage), with hands-on experience in designing, optimizing, and integrating scalable data pipelines and architectures across one or more of these platforms
• In-depth experience with Delta Lake, including schema enforcement, time travel, and optimization for ACID transactions
• Strong understanding of data pipelines, transformation workflows, and modeling for analytics and reporting, including scheduling, monitoring, and error handling in distributed data processing environments
• Experience in building automation tools or reusable scripts to accelerate data engineering processes
• Proven ability to lead technical workshops, gather complex requirements, and translate them into actionable solutions
• Strong communication skills to collaborate with clients, internal teams, and stakeholders at all levels
• Experience mentoring team members or supporting the professional growth of peers

Responsibilities

• Design and implement scalable, high-performance data architectures using Databricks and the Medallion Architecture
• Lead requirement-gathering sessions with clients and internal teams to understand business needs and define best practices for solutions including data workflows
• Collaborate with cloud platform teams to optimize data storage and retrieval in environments like AWS S3, Azure Data Lake, and Delta Lake
• Translate complex data processes, such as those in Alteryx and Tableau, into optimized Databricks workflows using PySpark and SQL
• Develop reusable automation scripts to streamline workflow migrations and improve operational efficiency
• Provide hands-on development and troubleshooting support to ensure smooth implementation and optimal performance
• Partner with cross-functional teams to establish data governance frameworks, best practices, and standardized reporting processes
• Deliver training, documentation, and ongoing support to empower users and enhance organizational data literacy
• Stay ahead of industry trends, identifying opportunities to integrate new tools and methods that enhance the practice's capabilities

Preferred Qualifications

• Background in supply chain, retail, or related industries
• Relevant certifications such as Databricks Certified Data Engineer Professional or equivalent

Benefits

• Comprehensive Health Insurance with generous employer contribution
• Matching 401(k) - $$$$
• Generous PTO Policy
• Virtual Events and Team Meetings
• Wellness Program