Hadoop Big Data Administrator/Engineer

Summary

Join Capco Poland, a global technology and management consultancy, as a Big Data Administrator/Engineer. This role requires 5+ years of Hadoop administration experience and expertise in automation tools. You will administer and manage Hadoop clusters, develop automation scripts, apply DevOps principles, and troubleshoot Big Data platform issues. The ideal candidate possesses strong DevOps skills, proficiency in Python, and a passion for Big Data technologies. Capco offers various benefits, including remote work possibilities, multiple employee benefit packages, access to online courses, and a supportive work culture.

Requirements

• 5+ years of hands-on experience in Big Data administration, automation, and DevOps practices, including Hadoop ecosystem tools such as HDFS, YARN, and MapReduce
• Proficient in managing and troubleshooting Kafka, HBase, and Spark
• Experience with performance tuning, cluster optimization, and high-availability configurations
• Strong experience with automation tools such as Ansible, Shell scripting, and Python scripting
• Ability to automate deployment, monitoring, and administration tasks to increase operational efficiency
• Solid understanding of DevOps concepts and practices, including CI/CD, version control, and deployment pipelines
• Proficiency in at least one programming language, with a preference for Python
• Ability to write efficient, maintainable code for automation and integration tasks
• Strong debugging skills and experience in resolving complex issues across distributed systems
• Ability to think critically and provide practical solutions in high-pressure situations
• Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience)

Responsibilities

• Administer and manage Hadoop clusters, ensuring high availability, performance, and scalability
• Maintain and troubleshoot Hadoop ecosystem components such as HDFS, MapReduce, YARN, and related tools
• Ensure Kafka, HBase, and Spark systems are optimized and running smoothly
• Implement monitoring and alerting for Big Data infrastructure
• Develop and automate scripts using tools such as Ansible, Shell scripting, or Python to streamline Big Data administration tasks
• Create reusable automation frameworks to reduce manual efforts and improve operational efficiency
• Work on CI/CD pipelines for deployment automation and system integration
• Apply DevOps principles to the Big Data environment, focusing on continuous integration and continuous delivery (CI/CD)
• Build and manage automated deployment processes for Big Data clusters and services
• Collaborate with development teams to integrate automation in Big Data workflows
• Identify, troubleshoot, and resolve issues related to Big Data platforms, including system performance, resource utilization, and service failures
• Work with logs, monitoring tools, and other debugging techniques to diagnose and resolve complex issues
• Work closely with other teams to support data pipelines, data quality checks, and performance optimizations
• Provide ongoing technical support to ensure that Big Data systems are stable, secure, and aligned with business objectives

Preferred Qualifications

• Experience with cloud platforms (AWS, Azure, GCP) for Big Data solutions
• Knowledge of containerization and orchestration tools (Docker, Kubernetes)
• Familiarity with data pipeline frameworks and tools like Apache NiFi, Airflow, or similar

Benefits

• Employment contract and/or Business to Business - whichever you prefer
• Possibility to work remotely
• Multiple employee benefits packages (MyBenefit Cafeteria, private medical care, life-insurance)
• Access to 3.000+ Business Courses Platform (Udemy)
• Access to required IT equipment
• Paid Referral Program
• Participation in charity events e.g. Szlachetna Paczka
• Ongoing learning opportunities to help you acquire new skills or deepen existing expertise