Senior Data Engineer

Summary

Join Sympower's data team as a Senior Data Engineer and focus on the technical integration of data within the company's Data Platform. You will build and enhance data integration pipelines, optimize data infrastructure, and ensure seamless data integration across software systems. Collaborate with analytics engineers to understand data requirements and optimize data structures. Automate data quality checks and troubleshoot pipeline issues. Integrate data from various internal and external sources. Develop batch and streaming data pipelines using the Databricks stack. Drive data architecture and modeling across the platform. Work with other data team members to deliver end-to-end data products. Sympower offers a competitive compensation package and a values-based culture.

Requirements

• Strong software-engineering mindset, approach and skills (Python)
• Experience with git, CI/CD, testing frameworks, DevOps
• Experience with building and improving data infrastructure, familiar with the Lake House concept and data cloud infrastructure
• Experience with data integration and processing: Big data storage, processing, ELT (batch/streaming) development and optimisation in Pyspark or other frameworks
• Knowledge of data modelling techniques, e.g. Kimball, Inmon, etc
• Advocate of data architecture standards: high-level infrastructure/pipeline design, data flow diagrams, data models
• Experience and affinity with mentoring and coaching other team members, helping them to grow
• Experience driving a project from start to end, including planning, task breakdown, discussions with stakeholders and working together with product management
• Challenging and steering the team’s technical and product roadmap

Responsibilities

• Design, develop, and maintain scalable data infrastructure and ETL pipelines to support self-service data solutions, ensuring data quality, reliability, and availability for end-users
• Collaborate with analytics engineers to understand data requirements and optimise data structures, enabling efficient querying and data transformations for analytics and reporting
• Automate data quality checks and data observability tools to monitor pipeline health and data accuracy, providing insights and alerts on data anomalies or pipeline failures
• Troubleshoot and resolve data pipeline issues quickly, providing support to analytics engineers and end-users to ensure minimal disruption to data workflows
• Integrate data from various internal and external sources from APIs, Apache Pulsar, (no)SQL databases
• Develop batch and streaming data pipelines with the full Databricks stack, with a strong focus on software engineering best practices
• Drive data architecture and coherent data modelling across our data platform
• You will work with the other roles in the data team (Machine Learning Engineers and Analytics Engineers) to deliver end-to-end data products to the other Sympower teams, the Sympower platform and our customers

Preferred Qualifications

• Fluent in English and other languages are nice to have
• Experience and affinity with our current data team’s tech stack: Python, Databricks, Spark, Delta Lake, Pulsar, ****Postgres, AWS

Benefits

• 30 Days Paid Holiday Leave
• 1 Day Paid Wellness Leave
• 1 Day Paid Birthday Leave
• Paid Maternity and Partner Leave
• Pawternity Leave
• Mental Health and Wellbeing Support
• Remote Office Budget
• Internet Allowance
• Development Plan & Budget
• Stock Appreciation Rights
• 2 Days Paid Volunteer Leave