Senior Data Engineer

Summary

Join Skillsoft's Enterprise Data Solutions team as a Sr. Data Engineer and contribute to building a robust data infrastructure. Lead the identification of business data requirements, design data models and ETL processes, and develop data products using Azure Databricks, SQL Server, and open-source tools. Collaborate with stakeholders to ensure alignment with business needs and work with the latest technologies to build efficient and scalable data solutions. Perform routine testing, monitor performance, and implement optimization strategies. Contribute to code reviews, provide technical guidance, and support the BI mission through learning new technologies. This role requires strong data engineering, data analysis, and business intelligence expertise, along with experience in Azure Databricks, SQL, and PySpark. Skillsoft offers a dynamic work environment and opportunities for professional growth.

Requirements

• Bachelor’s degree in quantitative field – engineering, finance, data science, statistics, economics, or other quantitative
• 5+ years of experience in Data Engineering/Data Management space and working with enterprise level production data warehouses
• 5+ years of experience in working with Azure Databricks
• 5+ years experience in SQL and PySpark
• Ability to work in an Agile methodology environment
• Experience and interest in cloud migration/journey to the cloud for data platforms and landscape
• Strong business acumen, analytical skills, and technical abilities
• Practical problem-solving skills and ability to move complex projects forward

Responsibilities

• Lead the identification of business data requirements, create data models and design processes that align to the business logic and regularly communicate with business stakeholders to ensure delivery meets business needs
• Design ETL processes, develop source-to-target mappings/integration workflows and manage load processes to support regular and ad hoc activities considering the needs of down-stream systems, functions and visualizations
• Work with the latest open-source tools, libraries, platforms and languages to build data products enabling other analysts to explore and interact with large and complex data sets
• Build robust systems and reusable code modules to solve problems across the team and organization with an eye on the long-term maintenance and support of the application
• Perform routine testing of own and others’ work to guarantee accurate, complete processes that support business needs. Awareness and compliance with all organizational development standards, industry best practices and business, security, privacy, and retention requirements
• Routinely monitor performance, diagnose and implement tuning/optimization strategies to guarantee a highly efficient data structure
• Collaborate with other engineers through active participation in code reviews and challenge the team to deliver with precision, consistency and speed
• Document data flows and technical designs to ensure compliance with organization, business and security best practices
• Regularly monitor timelines and workload. Ensure delivery promises are met or exceeded
• Ability and willingness to support the BI mission through learning new technologies and supporting other projects as needed
• Provides code reviews and technical guidance to the team
• Collaborate closely with the SA and TPO and get the requirements and develop the enterprise solutions