Python Crawler Engineer

Summary

The job is for a Data Engineer at a globally recognized digital asset exchange, Gate.io. The candidate should have a bachelor's degree or higher, be proficient in Python, scrapy, requests, Mysql/MongoDB/Redis, JS, and have experience with high parallelism and large data processing. Good communication skills, self-motivation for learning, and a problem-solving mindset are also required.

Requirements

• Bachelor's degree or higher
• 3 years or more experience in data business
• Proficient in Python
• Familiar with scrapy, requests, etc. scraping frameworks and HTTP tools
• Familiar with Mysql/MongoDB/Redis
• Familiar with JS, common anti-crawling mechanisms
• Experience with high parallelism and large data processing is preferred

Responsibilities

• Data collection
• Data cleaning
• System development