Senior Data Engineer

Summary

Join HubSpot's Employee Technology group as a Senior Data Engineer. You will design, build, and maintain data pipelines, working with various data sources and ensuring data security and efficiency. This role involves data acquisition and ingestion, AWS infrastructure management, data migrations, and data modeling. You will leverage technologies like Snowflake, AWS Lambda, and dbt. Your contributions will directly impact HubSpot's data-driven decision-making. The role offers a competitive salary and benefits package.

Requirements

• Proven experience in developing custom data ingestion pipelines and integrating with APIs, effectively managing complexities such as rate limiting and pagination
• Proficient in automating workflows using AWS Lambda, Step Functions, or comparable tools
• Extensive knowledge of databases (especially Snowflake.) Experience with Snowflake technologies such as Snowpipe, Snowpark, Data Sharing, Cloning etc., and Snowpipe usage
• Strong background in SQL for data warehouse transformations
• Solid understanding of data warehousing principles, DAG structures, and comprehensive data validation/testing frameworks
• Skilled in Python for creating utility scripts and automated workflows
• Experience with data manipulation libraries such as Pandas, PySpark, or similar technologies
• Familiarity with streaming data solutions, including Kafka, AWS Kinesis, RabbitMQ, etc

Responsibilities

• Acquire data from multiple sources using pre-built connectors and custom solutions
• Develop data ingestion pipelines to effectively manage internal and external API interactions, addressing challenges such as rate limiting, pagination, and retry mechanisms
• Process and securely ingest large datasets, ensuring encrypted data transfer and runtime credential safety
• Implement Snowpipe for data ingestion into Snowflake and utilize Snowflake’s external tables for accessing S3 data
• Manage and optimize data workflows within the AWS ecosystem, with a focus on S3 operations
• Manipulate S3 files upon arrival, archive older files to Glacier, and do light cleansing/alteration of files, as needed, for various workflows
• Automate AWS processes using tools like AWS Lambda, Step Functions, or custom scripts to improve efficiency and reliability
• Develop scripts for data migrations between Snowflake instances and automate tasks using Looker’s APIs
• Create utility scripts for business needs, such as disabling inactive users, purging broken/unused Looker content, and simplifying otherwise burdensome/manual processes
• Build and maintain data models using dbt, following best SQL practices and data warehousing principles to ensure schema design and data integrity
• Develop data pipelines in a DAG structure, ensuring clear dependencies and performing data validation and testing
• Implement CI/CD workflows with GitHub actions/hooks for automated testing and deployment

Preferred Qualifications

Experience with dbt

Benefits

• Cash compensation range: 157600-236400 USD Annually
• Base salary
• On-target commission for employees in eligible roles
• Annual bonus targets under HubSpot’s bonus plan for eligible roles
• HubSpot’s equity plan to receive restricted stock units (RSUs)
• Overtime pay