Remote Data Engineer

Summary

Join the CDC Foundation as a Data Engineer to design, build, and maintain data infrastructure for a public health organization. This role will play a crucial part in advancing the CDC Foundation's mission by delivering a transition plan and architecture needed for migrating portions of the Integrated Infectious Disease Data Warehouse (I2D2) infrastructure from one cloud-based platform to another.

Requirements

• Bachelor's degree in Computer Science, Information Technology, Data Science, or a related field
• Minimum 5 years of relevant professional experience
• Proficiency in programming languages commonly used in data engineering, such as Python, Java, Scala, SQL, or R
• Experience with Microsoft Azure products
• Strong understanding and experience with cloud-based warehouses/data lakes including Snowflake and Databricks
• Experience regarding engineering best practices such as source control, automated testing, continuous integration and deployment, and quality checks
• Knowledge of data warehousing concepts and tools
• Experience with cloud computing platforms
• Expertise in data modeling, ETL (Extract, Transform, Load) processes, and data integration techniques
• Familiarity with agile development methodologies, software design patterns, and best practices
• Strong analytical thinking and problem-solving abilities
• Excellent verbal and written communication skills, including the ability to convey technical concepts to non-technical partners effectively
• Flexibility to adapt to evolving project requirements and priorities
• Outstanding interpersonal and teamwork skills; and the ability to develop productive working relationships with colleagues and partners
• Experience working in a virtual environment with remote partners and teams
• Proficiency in Microsoft Office

Responsibilities

• Learn and assess the current cloud- based I2D2 infrastructure and functionalities
• Understand and assess current ETL processes to ensure the quality, accuracy, completeness, and consistency of data
• Collaborate with cross-functional teams to understand which I2D2 functions and business needs are supported by CDPH’s Databricks instance
• Collaborate with informatics specialists, enterprise architects, and other IT staff to understand data needs and requirements and ensure that the proposed data infrastructure supports the organization's security and privacy standards
• Partner with ITSD, IB, CalREDIE Branch, and CalCONNECT Branch to outline a comprehensive transition plan that meets the needs of all teams without sacrificing necessary functionality
• Develop the foundational framework for establishing a Center-wide data lake within the databricks platform to optimize data pipelines, infrastructure, and workflows for performance and scalability
• Identifying specific deliverables and expectations for the transition by scheduled dates and timelines
• Knowledgeable about industry trends, best practices, and emerging technologies in data engineering, and incorporating the trends into the organization's data infrastructure
• Provide technical guidance to other staff
• Communicate effectively with partners at all levels of the organization to gather requirements, provide updates, and present findings