Data Systems Engineer

Summary

Join Edge & Node, a leader in web3 innovation, as a Data Systems Engineer. You will contribute to the development of cutting-edge data services for processing blockchain data, leveraging Apache DataFusion and building data transformation pipelines. This role requires experience in database system internals and a rigorous approach to software development. Rust familiarity is highly desirable. The position offers a fully remote work environment and a competitive compensation package within a market range of $100k-$500k annually.

Requirements

• Have experience in database system internals
• Demonstrate adaptability to a dynamic and fully-remote work environment
• Show a rigorous approach to software development that reflects a commitment to excellence

Responsibilities

• Leverage and contribute to Apache DataFusion, closely monitoring upstream developments and becoming familiar with its codebase
• Implement storage strategies for blockchain and other web3 data sources, while leveraging industry standard formats and keeping in mind their tradeoffs
• Develop the internals of a streaming processing system. Given the streaming nature of blockchains, understand incremental techniques to update outputs with each new block
• Apply resource isolation in a multi-tenant environment for resources such as CPU, RAM and storage
• Build data transformation pipelines specifically for blockchain data and implement operations like aggregations and joins for streamlined usage
• Optimize compaction and garbage collection processes for performance and resource efficiency particularly in multi-tenant and high-throughput database systems
• Develop supportive tooling, including tracing and profiling

Preferred Qualifications

• Have Rust familiarity
• Have knowledge of Apache DataFusion and the related ecosystem
• Have general knowledge of distributed data systems and resource management
• Have background with data ingestion from blockchain or other high-throughput sources

Benefits

• Fully remote work environment
• Compensation in the range of $100k - $500k annually (total compensation)