Remote Sr Big Data Engineer

Summary

The job is for a Senior Big Data Engineer to work remotely. The candidate should have extensive experience with Oozie, the Apache Hadoop ecosystem, Airflow, GCP, Java, Python, Pig, SQL, and other related technologies. They should also be experienced in Infrastructure and Applied DevOps principles.

Requirements

• Bachelor's degree in Computer Science, software engineering or related field of study
• Experience with GCP managed services and understanding of cloud-based batch processing systems are critical
• Proficiency in Oozie, Airflow, Map Reduce, Java
• Strong programming skills with Java (specifically Spark), Python, Pig, and SQL
• Expertise in public cloud services, particularly in GCP
• Proficiency in the Apache Hadoop ecosystem with Oozie, Pig, Hive, Map Reduce
• Familiarity with BigTable and Redis
• Experienced in Infrastructure and Applied DevOps principles in daily work. Utilize tools for continuous integration and continuous deployment (CI/CD), and Infrastructure as Code (IaC) like Terraform to automate and improve development and release processes
• Ability to tackle complex challenges and devise effective solutions. Use critical thinking to approach problems from various angles and propose innovative solutions
• Worked effectively in a remote setting, maintaining strong written and verbal communication skills. Collaborate with team members and stakeholders, ensuring clear understanding of technical requirements and project goals
• Proven experience in engineering batch processing systems at scale
• Hands-on experience in public cloud platforms, particularly GCP. Additional experience with other cloud technologies is advantageous

Responsibilities

• Develop scalable and robust code for batch processing systems
• Develop, Manage and optimize data workflows using Oozie and Airflow within the Apache Hadoop ecosystem
• Leverage GCP for scalable big data processing and storage solutions
• Implementing automation/DevOps best practices for CI/CD, IaC, etc

Preferred Qualifications

• Google Associate Cloud Engineer Certification or other Google Cloud Professional level certification
• 5+ years of experience in customer-facing software/technology or consulting
• 5+ years of experience with ‘on-premises to cloud’ migrations or IT transformations