Mid/Senior Data Engineer

Summary

Join Capco Poland, a global technology and management consultancy, as a Senior Big Data Engineer. This remote position, with potential for business trips, involves designing, implementing, and maintaining scalable data pipelines and solutions. You will work with clients, design robust data pipelines, and adhere to best practices. Leadership and mentoring of junior engineers are key aspects, along with contributing to security designs. The ideal candidate possesses extensive experience in various technologies and methodologies, including cloud platforms (GCP preferred), Python, SQL, and big data technologies (Hadoop, Spark, etc.). Capco offers a competitive compensation package, including remote work options, multiple employee benefits, and ongoing learning opportunities.

Requirements

• Strong cloud provider’s experience on GCP
• Hands on experience using Python
• Experience in most or all of data and cloud technologies such as Hadoop, HIVE, Spark, Flume, PySpark, DataProc, Cloudera, Airflow, Oozie, S3, Terraform etc
• Hands on experience with schema design using semi-structured and structured data structures
• Experience using messaging technologies – Kafka, Kafka Connect, Spark Streaming, Amazon Kinesis
• Strong experience in SQL
• Good understanding of the differences and tradeoff between SQL and NoSQL, ETL and ELT
• Understanding of containerisation (Docker, Kubernetes) and orchestration techniques
• Experience with data lake formation and data warehousing principles and technologies – BigQuery, Redshift, Snowflake
• Experience using version control tool such as Git
• Familiar with development good practices and optimisation techniques
• Experience in design, build and maintain CI/CD Pipelines on Jenkins, CircleCI
• Enthusiasm and ability to pick up new technologies as needed to solve your problems

Responsibilities

• Work alongside clients to interpret requirements and define industry-leading solutions
• Design and develop robust, well tested data pipelines
• Demonstrate and help clients adhere to best practices in engineering and SDLC
• Build and improve strong relationships with peers, senior stakeholders and client
• Leading and mentoring the team of junior and mid-level engineers
• Contribute to security designs and have advanced knowledge of key security technologies e.g. TLS, OAuth, Encryption
• Support internal Capco capabilities by sharing insight, experience and credentials

Preferred Qualifications

• Scala
• GCP
• Pub/Sub
• Big Query
• Kafka
• Juniper
• Apache NiFi
• Hive
• Impala
• Cloudera
• CI/CD
• Scala and Java

Benefits

• Employment contract and/or Business to Business - whichever you prefer
• Possibility to work remotely
• Multiple employee benefits packages (MyBenefit Cafeteria, private medical care, life-insurance)
• Access to 3.000+ Business Courses Platform (Udemy)
• Access to required IT equipment
• Paid Referral Program
• Participation in charity events e.g. Szlachetna Paczka
• Ongoing learning opportunities to help you acquire new skills or deepen existing expertise