Senior Data Engineer

Summary

Join Profasee, a data engineering team, as a Python Engineer to revolutionize the e-commerce industry. You will collaborate with e-commerce experts and the data science/machine learning team to build, test, and maintain data systems. Responsibilities include building data pipelines, storage systems, and APIs, providing support to the ML engineering team, and ensuring data quality. The ideal candidate possesses strong Python and SQL skills, experience with ETL pipelines, and a knack for building well-tested, readable code. This is a fully remote position within a dynamic, entrepreneurial startup environment.

Requirements

• 5+ years of software development experience
• Proficient with Python programming language
• Proficient with SQL programming language and databases (PostgreSQL, MySQL, etc)
• Proficient integrating with APIs, data formats (JSON, CSV, XML, etc), rate limiting, and error handling
• Experience with distributed task queues (i.e. Celery), multiprocessing, job scheduling
• Understanding of the infrastructure used to build a production application (ex. NGINX, RabbitMQ, Error and Performance monitoring tools, AWS services such as ECS/EKS, etc.)
• Strong communication skills, especially with non-software developers
• Experience with containerization (Docker/Kubernetes)
• Experience designing and delivering the architecture required to run code in production
• Experience with ETL pipelines

Responsibilities

• Work side by side with e-commerce experts and the DS / ML team to build, test, and maintain systems that collect, manage, and convert raw data into usable information for the DS / ML team to interpret
• Be responsible for building and maintaining data pipelines, storage systems, and APIs to access the data
• Provide support to the ML engineering team to help with data engineering
• Write code and tests for pulling data from 3rd party integrations and loading into a structured database for each data source (ETL)
• Provide API access to the data collected from 3rd parties
• Analyze data sources to help decide which are useful for the DS / ML team
• Maintain the data pipeline architecture, automation, scalability, and error handling
• Monitor pipeline performance and stability

Preferred Qualifications

• Experience working in early-stage startups and/or developing your own projects
• Experience with NoSQL and non-relational databases
• Experience with machine learning and data science