Remote Senior Data Engineer

Summary

The job is for a Senior Data Engineer to work remotely full-time for a Swedish client, supporting the Data Intelligence team by creating new solutions, optimizing existing pipelines, and managing big data. The candidate should have at least 8 years of experience in data engineering or a similar role, with expertise in Microsoft SQL Server, Azure (ADF, Blob Storage), .NET (C#), DataBricks, Spark, NATS Jetstream, Docker/Kubernetes, Oracle DB, and big data systems like DataBricks, Lakehouse, Kafka.

Requirements

• At least 8 years of experience in the role of a data engineer or a similar position
• Experience with Microsoft SQL Server, Microsoft Azure (ADF, Blob Storage), .NET (C#), DataBricks, Spark, NATS Jetstream, Docker/Kubernetes
• Advanced knowledge of SQL and relational databases, preferably Oracle DB
• Experience with data queuing systems, streaming processing, and big data storage systems (DataBricks, Lakehouse, Kafka)
• Experience in building and optimizing big data pipeline's
• Ability to perform root cause analysis
• Strong analytical skills and experience with unstructured data

Responsibilities

• Create and maintain data pipelines and collaborate in maintaining and optimizing the data architecture
• Build and optimize complex data sets that meet business requirements, both functional (e.g., fast search and filter of data, synchronization with internal systems) and non-functional (e.g., data storage according to legal regulations)
• Design and implement internal process improvements (e.g., automation of manual processes, optimization of data delivery)
• Build the infrastructure necessary for optimizing the ETL process from various sources using SQL and cloud technologies
• Collaborate with architects and designers to unify existing data models, pipelines, data storages
• Create analytical tools and microservices using data pipelines, supporting the work of data analysts
• Collaborate with stakeholders, including engineering management, product, data, and design teams to solve technical problems and support their infrastructure needs
• Build processes supporting data transformation, data structure, metadata management, dependency/workload management

Preferred Qualifications

• Knowledge of Python
• Experience in working in the AWS environment