Remote Data Ingestion Engineer, Senior

Important Information

Location: Brazil Job Mode: Full-time  Work Mode: Work from home

Job Summary

This role will be responsible for conducting data ingestion and platform management on the Databricks platform. This role requires a deep understanding of: Data Lake ingestion processes and best practices; ETL/ELT implementation; CI/CD; System integration tools; Data pipeline management.

Responsibilities and Duties

• Ingest data from a variety of source systems and tailor ingestion approaches on a per-system basis;

• Manage, maintain, and oversee ETL/ELT pipelines on the Databricks platform;

• Optimize data pipelines for scalability and speed;

• Document ingestion and integration flows and pipelines;

• Use Airflow to schedule and automate ingestion jobs;

• Manage metadata and master data in the technical data catalog;

• Ensure ELT/ETL design meets required security and compliance guidelines, and ensure PII management, flagging and risk assessment during ingestion;

• Maintain ETL/ELT pipeline infrastructure and implement automated monitoring strategies;

• Ensure adherence to SDLC best practices.

Essential Skills

• Experience in data engineering, ingestion pipelining, and ETL/ELT;

• Hold a bachelor’s degree in computer science, engineering, statistics, or related field;

• Have hands-on experience with and understanding of the following:

 - Spark/Scala;

 - SQL;

 - Python/PySpark or similar programming language;

 - Databricks;

 - Unity Catalog;

 - ETL/ELT development, monitoring, and pipelining using tools such as Apache Airflow;

 - Ingestion tools such as Dell Boomi;

 - Data quality guidelines;

 - CI/CD pipelines;

 - Agile;

 - Git and version control.

About Encora

Encora is the preferred digital engineering and modernization partner of some of the world’s leading enterprises and digital native companies. With over 9,000 experts in 47+ offices and innovation labs worldwide, Encora’s technology practices include Product Engineering & Development, Cloud Services, Quality Engineering, DevSecOps, Data & Analytics, Digital Experience, Cybersecurity, and AI & LLM Engineering.

At Encora, we hire professionals based solely on their skills and qualifications, and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.