Remote Data Ingestion Engineer, Senior
Encora
Job highlights
Summary
Join our team as a Data Engineer to conduct data ingestion and platform management on the Databricks platform. This role requires expertise in data engineering, ingestion pipelining, and ETL/ELT, with experience in Spark/Scala, SQL, Python/PySpark, and Databricks.
Requirements
- Experience in data engineering, ingestion pipelining, and ETL/ELT
- Hold a bachelor’s degree in computer science, engineering, statistics, or related field
- Spark/Scala
- SQL
- Python/PySpark or similar programming language
- Databricks
- Unity Catalog
- ETL/ELT development, monitoring, and pipelining using tools such as Apache Airflow
- Ingestion tools such as Dell Boomi
- Data quality guidelines
- CI/CD pipelines
- Agile
- Git and version control
Responsibilities
- Ingest data from a variety of source systems and tailor ingestion approaches on a per-system basis
- Manage, maintain, and oversee ETL/ELT pipelines on the Databricks platform
- Optimize data pipelines for scalability and speed
- Document ingestion and integration flows and pipelines
- Use Airflow to schedule and automate ingestion jobs
- Manage metadata and master data in the technical data catalog
- Ensure ELT/ETL design meets required security and compliance guidelines, and ensure PII management, flagging and risk assessment during ingestion
- Maintain ETL/ELT pipeline infrastructure and implement automated monitoring strategies
- Ensure adherence to SDLC best practices
Job description
Important Information
Location: Brazil Job Mode: Full-time Work Mode: Work from home
Job Summary
This role will be responsible for conducting data ingestion and platform management on the Databricks platform. This role requires a deep understanding of: Data Lake ingestion processes and best practices; ETL/ELT implementation; CI/CD; System integration tools; Data pipeline management.
Responsibilities and Duties
Ingest data from a variety of source systems and tailor ingestion approaches on a per-system basis;
Manage, maintain, and oversee ETL/ELT pipelines on the Databricks platform;
Optimize data pipelines for scalability and speed;
Document ingestion and integration flows and pipelines;
Use Airflow to schedule and automate ingestion jobs;
Manage metadata and master data in the technical data catalog;
Ensure ELT/ETL design meets required security and compliance guidelines, and ensure PII management, flagging and risk assessment during ingestion;
Maintain ETL/ELT pipeline infrastructure and implement automated monitoring strategies;
Ensure adherence to SDLC best practices.
Essential Skills
Experience in data engineering, ingestion pipelining, and ETL/ELT;
Hold a bachelor’s degree in computer science, engineering, statistics, or related field;
Have hands-on experience with and understanding of the following:
- Spark/Scala;
- SQL;
- Python/PySpark or similar programming language;
- Databricks;
- Unity Catalog;
- ETL/ELT development, monitoring, and pipelining using tools such as Apache Airflow;
- Ingestion tools such as Dell Boomi;
- Data quality guidelines;
- CI/CD pipelines;
- Agile;
- Git and version control.
About Encora
Encora is the preferred digital engineering and modernization partner of some of the world’s leading enterprises and digital native companies. With over 9,000 experts in 47+ offices and innovation labs worldwide, Encora’s technology practices include Product Engineering & Development, Cloud Services, Quality Engineering, DevSecOps, Data & Analytics, Digital Experience, Cybersecurity, and AI & LLM Engineering.
At Encora, we hire professionals based solely on their skills and qualifications, and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.
Share this job:
Similar Remote Jobs
- 💰$225k-$255k📍United States
- 💰$160k-$180k📍Worldwide
- 📍Worldwide
- 📍India
- 📍Worldwide
- 💰$165k-$180k📍United States
- 💰$72k-$110k📍Canada
- 📍Worldwide