Data Engineer, Python Senior

Summary

Join Encora as a Data Engineer to work on chatbot and copilot applications. You will lead the design and implementation of an efficient RAG system, integrating multiple data sources. This role requires strong Python programming skills and experience with embedding models, vector databases, and similarity search techniques. Experience with LLM frameworks and data preprocessing techniques is essential. The position is full-time and remote, located in Brazil.

Requirements

• Strong proficiency in Python programming with experience in building scalable and maintainable codebases
• Solid understanding of embedding models, vector databases, and similarity search techniques
• Hands-on experience with LLM frameworks such as LangChain
• Knowledge of data preprocessing techniques for handling textual, audio, and video data
• Data and ML Knowledge
• Experience with file ingestion processes for diverse formats (e.g., text, graphics, tables, images, PPT, video)
• Experience with RAG (Retrieval-Augmented Generation) workflows and their integration with LLMs
• Understanding of vector mathematics, including cosine similarity and Euclidean distance

Responsibilities

• Implement the ingestion and integration of multiple data sources (audio, video, PowerPoint presentations, documents, etc.)
• Design and implementation of data structure within appropriate storage solutions
• Enhance data quality through version control and updates
• Implement efficient search algorithms to optimize data retrieval
• Participate in the design of an efficient RAG system
• Hands-on programming using Python

Preferred Qualifications

Practical expertise with tools like FAISS, Pinecone, Weaviate, or similar