Big Data Engineer

Summary

Join our dynamic team as a Mid-Senior Big Data Engineer and collaborate with Solution Architects to develop innovative, scalable solutions for our customers. You will work directly with clients in foreign countries, ensuring high-quality solutions and seamless collaboration. This role involves developing and implementing Big Data solutions using various technologies, leveraging data visualization tools, and applying knowledge of real-time data ingestion. You will work within the Hadoop ecosystem, design ETL/ELT pipelines, and collaborate with internal teams to showcase Huawei Cloud Big Data capabilities. Troubleshooting and optimizing batch processing workflows are also key responsibilities. The ideal candidate will possess strong technical skills and excellent communication abilities.

Requirements

• BSc or MSc degree in Computer Engineering, Computer Science, Software Engineering, or a related technical field
• Minimum of 3 years of professional experience in Big Data engineering, with hands-on experience in Spark, Flink, Hadoop, and related technologies
• Have a knowledge in Python or Java programming languages
• Have an experience with SQL development (MySQL, PostgreSQL)
• Hands-on experience with batch processing using Hive, Spark
• Knowledge of Data Storage Formats (ORC, Parquet, CSV)
• Have a knowledge in Data Visualization tools (PowerBI, Tableau, Grafana) for building meaningful insights from complex data
• Experience with Data Warehousing and Data Lakes concepts
• Ability to communicate effectively and present technical concepts to both technical and non-technical audiences
• Experience working in Unix/Linux environments
• Fluency in written and spoken English is a must
• Enthusiasm for continuous learning and sharing knowledge with colleagues

Responsibilities

• Develop and implement Big Data solutions using technologies such as Python or Java, and SQL
• Support and collaborate with Solution Architects to develop optimal customer architecture solutions
• Leverage Data Visualization tools (PowerBI, Tableau, Grafana) to build meaningful insights from data analysis scenarios
• Apply knowledge of real-time data ingestion technologies to design efficient data flows
• Work within the Hadoop ecosystem (HDFS, MapReduce, Hive, Spark) to process and analyze datasets
• Design, implement, and maintain ETL/ELT pipelines
• Work with data storage formats such as ORC, Parquet, and CSV
• Collaborate with internal teams to explain and demonstrate Huawei Cloud Big Data capabilities
• Engage with customers in foreign countries, providing on-site or remote support and ensuring high-quality customer service and solution implementation
• Troubleshoot and optimize batch processing workflows using Hive, Spark, and other Big Data technologies

Preferred Qualifications

• Solid understanding of ETL/ELT methodologies and data processing best practices
• Designing and implementing complex pipelines in Data Warehouse and Data Lakes
• Comprehensive knowledge on open source Big Data product management. ( Hadoop(Flink,Hive,Spark, Clickhouse))
• Comprehensive knowledge about Apache Hudi, Iceberg