Data Engineer, Python Senior

Summary

Join Encora as a Data Engineer and play a key role in developing and optimizing data pipelines for LLM/RAG systems. You will integrate diverse data sources, implement advanced search algorithms, and design scalable data architectures. Collaborating with cross-functional teams, you will ensure efficient data ingestion, transformation, and retrieval. This full-time, work-from-home position in Brazil requires strong Python programming skills and a solid understanding of embedding models and vector databases. Experience with LLM frameworks and data preprocessing techniques is essential. The role involves hands-on programming and participation in the design of an efficient RAG system.

Requirements

• Strong proficiency in Python programming with experience in building scalable and maintainable codebases
• Solid understanding of embedding models, vector databases, and similarity search techniques
• Hands-on experience with LLM frameworks such as LangChain
• Knowledge of data preprocessing techniques for handling textual, audio, and video data
• Data and ML Knowledge
• ETL/ELT: Familiarity with data processing workflows or pipelines
• Experience with file ingestion processes for diverse formats (e.g., text, graphics, tables, images, PPT, video)
• Experience with RAG (Retrieval-Augmented Generation) workflows and their integration with LLMs
• Understanding of vector mathematics, including cosine similarity and Euclidean distance

Responsibilities

• Implement the ingestion and integration of multiple data sources (audio, video, PowerPoint presentations, documents, etc.)
• Design and implementation of data structure within appropriate storage solutions
• Enhance data quality through version control and updates
• Implement efficient search algorithms to optimize data retrieval
• Participate in the design of an efficient RAG system
• Hands-on programming using Python

Preferred Qualifications

Practical expertise with tools like FAISS, Pinecone, Weaviate, or similar is highly desirable

Benefits

Work from home