Lead Data Engineer

Summary

Join UpGuard's Analytics team and play a key role in democratizing data insights and analytics. Design, build, and maintain reliable data pipelines consolidating information from various internal and external sources. Develop and manage a comprehensive semantic layer, ensuring data quality, consistency, and reliability. Enable AI agents with access to necessary data for tasks like answering questions and identifying opportunities. Create clear documentation for data models and pipelines. This role is crucial for scaling UpGuard's analytics function to meet the growing needs of our expanding business.

Responsibilities

• Data Integration & Pipeline Development: Design, build, and maintain reliable data pipelines to consolidate information from various internal systems (CRM, support platforms, marketing automation, product usage data, etc.) and relevant third-party sources
• Semantic Layer Design & Management: Develop and manage a comprehensive semantic layer (potentially using technologies like LookML or similar) that provides clear definitions and mappings for data concepts to underlying database structures as well as high level business logic
• Data Quality & Governance: Implement and enforce data quality checks, validation rules, and governance processes to ensure the accuracy, consistency, and reliability of data
• AI & Agent Data Enablement: Ensure AI agents have access to the necessary structured and unstructured data (Knowledge Bases, support tickets, documentation, internal data, pricing info, marketing content, etc.) required for them to perform tasks like answering questions, qualifying leads, preparing for calls, and identifying opportunities
• Documentation & Knowledge Sharing: Create clear documentation for data models, pipelines, and the semantic layer to enable both human users and AI agents to effectively interact with the data warehouse

Preferred Qualifications

• Looker is our main BI platform: a candidate should be familiar with Explores, Looks, Dashboards and the Developer interface - maintaining dimensions and measures, create and maintain models, and be able to run and debug raw SQL queries. Be able to edit and deploy custom visualizations using Looker’s extension SDK. Interface with Looker’s API
• CloudSQL ( PostgreSQL ) and BigQuery for storage and query: be comfortable with writing complex queries, indices, materialised views, etc
• Containers, Docker and Kubernetes (via GKE) are used to run our internal automations. Being familiar and confident being a Platform Engineer in our GCP project
• Be familiar with technologies like n8n to automate processes and potentially Golang as a programming language as our non-standard ETL workers are written in Go
• Be comfortable interfacing with a variety of APIs from our vendor platforms (either REST+JSON or MCP Server, depending on what a vendor platform throws at you)
• Have experience with version control via GitHub , and a basic review process, like GitHub Flow
• If all else fails, being comfortable and confident falling back to a Golang + TypeScript + ChartJS solution, just to get data in the right shape or form to enable the Analyst to continue their work

Benefits

• Stock Options: share in our future success with stock options
• Monthly Lifestyle subsidy: use this for financial, physical and mental wellbeing
• WFH set-up allowance: to ensure you have the right environment to work in, we will help you get set-up within your first 3 months at UpGuard
• $1500 USD annual Learning & Development allowance: to support your career development all team members will be able to expense development opportunities against this allowance
• Generous Annual Leave/PTO allowances: time to recharge your batteries
• 18 weeks paid Parental Leave: irrespective of parenting role
• Personal Leave allowance: this includes sick & carer’s leave
• Fully remote working environment: whilst we have physical offices in Sydney & Hobart, we do not mandate compulsory attendance
• Top spec hardware: all team members will be provided with top-spec laptops for their role
• Generative AI subsidy: UpGuard provides paid subscriptions for all team members to access generative AI tools to support their work