Data Engineer

Summary

Join Experience.com as a Data Engineer and contribute to our impactful Experience Management Platform. This full-time, remote position requires 4+ years of experience. You will design, build, and maintain scalable ETL/ELT pipelines using PySpark and SQL, leveraging GCP data analytics tools. Responsibilities include optimizing data workflows, collaborating with engineering and analytics teams, and ensuring data quality. Strong proficiency in Python and experience with Airflow/Composer are essential. The ideal candidate will also possess familiarity with data pipeline mechanisms and experience with Elasticsearch and MongoDB.

Requirements

• 4+ years of experience with PySpark and SQL for building scalable ETL pipelines
• Strong proficiency in Python programming
• Knowledge of GCP Data Analytics ecosystem (BigQuery, PySpark, SQL, etc.)
• Experience with Airflow/Composer for workflow orchestration
• Experience with in-memory applications, database design, and data integration
• Strong analytical thinking and problem-solving abilities

Responsibilities

• Design, build, and maintain scalable and robust ETL/ELT pipelines using PySpark and SQL
• Work on data extraction, transformation, and loading processes from multiple sources into data warehouses such as BigQuery
• Leverage GCP data analytics tools (BigQuery, DataProc, Cloud Functions, etc.) to process and analyze data
• Optimize data workflows for benchmarking, performance, and tuning to ensure efficiency and reliability
• Collaborate with engineering and analytics teams to develop data integration solutions that meet business needs
• Ensure the accuracy and quality of data by implementing strong in-memory applications and database designs
• Implement monitoring and alerting for pipelines and workflows to ensure data consistency and issue resolution

Preferred Qualifications

• Familiarity with data pipeline mechanisms such as Kafka or Pub/Sub, Cloud Functions
• Hands-on experience with Elasticsearch, and MongoDB

Benefits

• Remote work
• Full Time Employment