Factored is hiring a
Data Engineer

Summary

The job description is for a Data Engineer position at Factored, an early-stage startup looking for an experienced professional to join their team. The role involves data pipeline management, data processing, ETL processes optimization, analytics tool management, API integration, infrastructure development using SQL and AWS big data technologies, and ensuring data security and compliance.

Requirements

• 3 - 5+ years of professional experience shipping high-quality, production-ready code
• Strong computer science foundations, including data structures & algorithms, OS, computer networks, databases, algorithms, and object-oriented programming
• Experience with Databricks
• Experience in Python and Java
• Experience in setting up data pipelines using relational SQL and NoSQL databases, including Postgres, Cassandra or MongoDB
• Experience with cloud services for handling data infrastructure such as: Snowflake(preferred), Azure, Databricks, Azure Databricks, and/or AWS
• Experience with orchestration tools such as Airflow
• Proven success manipulating, processing, and extracting value from large datasets
• Experience with big data tools, including Hadoop, Spark, Kafka, etc
• Expertise with version control systems, such as Git
• Strong analytic skills related to working with unstructured datasets
• Excellent verbal and written communication skills in English

Responsibilities

• Data aggregation, scraping, validation, transformation, quality and DevOps administration of both structured and unstructured datasets
• Convert data pipelines in Databricks to Unity Catalog and migrate their data storage practices to achieve Unity Catalog compatibility
• Build and operate data pipelines in Databricks to bring data from distributed storage locations to the central data lakehouse
• Create and maintain optimal data pipeline architecture across multiple data sources, including licensed and scraped data
• Assemble large, complex data sets that meet functional needs across Data Teams
• Design and develop optimal data processing techniques: automating manual processes, data delivery, data validation and data augmentation
• Develop any necessary ETL processes to optimize analysis and performance
• Manage analytics tools that provide actionable insights into usage, customer acquisition, operational efficiency and other key business performance metrics
• Design and develop a API integrations in order to feed different data models
• Architect and implement new features from scratch, partnering with AI/ML engineers to identify data sources, gaps and dependencies
• Identify bugs and performance issues across the stack, including performance monitoring and testing tools to ensure data integrity and quality user experience
• Build a highly scalable infrastructure using SQL and AWS big data technologies

Preferred Qualifications

• BSc in Computer Science, Mathematics or similar field; Master’s or PhD degree is a plus
• Experience with real-time scenarios, low-latency systems and data intensive environments is a plus
• Experience developing scalable RESTful APIs
• Experience with consumer applications and data handling

Benefits

• Transparent workplace, where everybody has a voice in building OUR company, and where learning and growth is available to everyone based on their merits, not just on stamps on their resume
• Investment in you and support for your career and professional growth in many meaningful ways