Data Engineer

Summary

Join PayPay's Data Insights department as a Data Engineer and help scale our platform. You will design, develop, and maintain scalable data ingestion pipelines using AWS services. Optimize and manage large-scale data pipelines for high performance and reliability. Implement data processing workflows using various tools and technologies. Collaborate with cross-functional teams and implement best practices for data governance and security. This is a full-time, work-from-anywhere position in Japan with flexible hours and comprehensive benefits.

Requirements

• 5+ years experience as a Data Engineer or in a similar role
• Hands-on experience with Apache Hudi, Delta Lake, Spark, and Scala
• Experience designing, building, and operating a DataLake or Data Warehouse
• Strong expertise in AWS services, including Glue, Step Functions, Lambda, and EMR
• Proficiency in Python and SQL (any variant), preferably experience in Scala and/or Java
• Experience with data cataloging and metadata management using AWS Glue Data Catalog, Lakeformation, or Unity Catalog
• Proficiency in Terraform for infrastructure as code (IaC)
• Overall understanding of machine learning technologies and deep learning concepts
• Strong problem-solving skills and ability to troubleshoot complex data issues
• Excellent communication and collaboration skills
• Ability to work in a fast-paced, dynamic environment and manage multiple tasks simultaneously

Responsibilities

• Design, develop, and maintain scalable data ingestion pipelines using AWS Glue, Step Functions, Lambda, and Terraform
• Optimize and manage large scale data pipelines to ensure high performance, reliability, and efficiency
• Implement data processing workflows using Hudi, Delta Lake, Spark, and Scala
• Maintain and enhance Lakeformation and Glue Data Catalog for effective data management and discovery
• Design, build, and maintain infrastructure to continuously support the improvement and deployment of machine learning models
• Collaborate with cross-functional teams to ensure seamless data flow and integration across the organization
• Implement best practices for observability, data governance, security, and compliance

Preferred Qualifications

• Knowledge of Data Orchestration tools such as Airflow, Dagster, Prefect
• Familiarity with change data capture tools like Canal, Debezium, and Maxwell
• Experience with data warehousing tools like AWS Athena, BigQuery, Databricks

Benefits

• Social Insurance (health insurance, employee pension, employment insurance and compensation insurance)
• 401K
• Translation/Interpretation support
• VISA sponsor + Relocation support
• Work From Anywhere at Anytime
• Super Flex Time (No Core Time)
• Every Sat/Sun/National holidays (In Japan)/New Year's break/Company-designated Special days
• Annual leave (up to 14 days in the first year, granted proportionally according to the month of employment. Can be used from the date of hire)
• Personal leave (5 days each year, granted proportionally according to the month of employment）
• PayPay's own special paid leave system, which can be used to attend to illnesses, injuries, hospital visits, etc., of the employee, family members, pets, etc
• Annual salary paid in 12 installments (monthly)
• Reviewed once a year
• Special Incentive once a year *Based on company performance and individual contribution and evaluation
• Late overtime allowance, Work from anywhere allowance (JPY100,000)