Senior Data Engineer

Summary

Join Abacus Insights as a Sr. Data Engineer to contribute to the success of our dynamic and rapidly expanding Tech Ops division, collaborating with customers, data vendors, and internal teams to integrate data into a state-of-the-art data platform.

Requirements

• Bachelor’s Degree in computer science, computer engineering or any related IT field
• Experience communicating with both technical and non-technical stakeholders
• Proficient with Python and SQL
• Build and manage efficient ETL pipelines using Databricks workflows or other orchestration framework
• Familiarity with both structured and semi-structured data and ingesting and processing this data using pyspark
• Fundamental AWS services (S3, SQS) or similar services for other clouds
• Terraform and GitLab CICD
• Query tuning and performance optimization in SQL and/or SparkSQL
• Familiarity with data warehousing (snowflake or similar)

Responsibilities

• Work with clients & implementation team and understand the data distribution requirements
• Develop data pipelines integrating with Databricks and Snowflake, ensuring data accuracy and integrity
• Lead the architecture and implementation of solutions for health plan clients, optimizing cloud-based technologies
• Collaborate with cross-functional teams and clients to translate requirements into technical solutions
• Implement robust data security measures and ensure compliance with healthcare regulations like HIPAA
• Provide technical leadership, mentoring junior engineers and staying updated on industry trends
• Focus on optimizing data processes for efficiency, regularly reviewing architectures, and creating detailed documentation
• Conduct training for team members and clients to enhance understanding and utilization of data solutions