Rackspace Technology is hiring a
Senior Data Engineer, Remote - Vietnam

Summary

The job is for a Senior Data Engineer or Data Architect with extensive experience in Spark Streaming, Azure Functions, Synapse, and other Azure data services. The role involves designing, developing, and maintaining real-time data streaming pipelines, implementing CI/CD pipelines, and collaborating with cross-functional teams to deliver high-quality data solutions.

Requirements

• Proven experience in real-time data streaming using Spark Streaming or Azure Functions
• Strong proficiency in Python for data processing and automation tasks
• Hands-on experience with Kafka for data ingestion and message queuing
• Familiarity with Redis for caching and fast data retrieval
• Knowledge of data lake architectures and best practices for data storage and retrieval
• Solid understanding of software engineering principles, including design patterns, testing, and documentation
• Strong problem-solving skills and ability to work in a fast-paced, collaborative environment

Responsibilities

• Design, develop, and maintain data architectures using Spark and Lakehouse methodologies
• Utilize Synapse services including Spark pools, dedicated SQL pools, and pipelines to build robust data solutions
• Leverage other Azure data services to enhance and optimize data workflows
• Implement and manage CI/CD pipelines to ensure smooth deployment and integration processes
• Load, merge, and process machine logs from Kafka, ensuring efficient data flow and transformation
• Integrate processed data into Redis cache and send it to the data lake for long-term storage and analysis
• Implement and optimize data processing solutions using Python
• Apply software engineering best practices, including code reviews, version control, and continuous integration/deployment (CI/CD)

Preferred Qualifications

• Experience with cloud platforms such as Azure
• Knowledge of other programming languages and frameworks related to data engineering
• Familiarity with data modeling, ETL processes, and data lake concepts