Senior Data Engineer

Summary

Join CodaMetrix as a Senior Data Engineer and contribute to our AI-powered revenue cycle management platform. You will design, develop, and manage data systems and architecture for processing and analyzing large datasets. Responsibilities include building scalable data pipelines, integrating data sources, ensuring data quality, and collaborating with cross-functional teams. You will also optimize data processing workflows, work with various data formats, and mentor junior engineers. The ideal candidate possesses a strong background in big data technologies, experience with Databricks and Spark, and a passion for data engineering best practices. This role offers a competitive salary and benefits package, including health insurance, retirement plan, paid time off, and professional development opportunities.

Requirements

• Required BS, MS degree in Computer Science, Informatics, Information Systems or other related fields or equivalent work experience
• 5+ years of working experience with Databricks platform using PySpark/Scala
• 5+ years of experience with big data technologies on data ingestion using Apache Spark, Apache Kafka, and other distributed computing tools
• 5+ years of strong SQL experience on relational and non-relational databases (SQL, NoSQL, MongoDB, etc.)
• Experience with object-oriented or functional programming languages: Scala, Java, and Python are all preferred
• Experience with both structured and unstructured data formats such as Parquet, CSV, JSON, XML
• Experience working with Terraform to provision cloud infrastructure
• Experience with GitHub for version control, collaborative development, and CI/CD pipelines
• Hands-on experience building and managing data pipelines in large-scale, cloud-based environments
• Good knowledge of BI Tools; Tableau is a huge plus
• Agile Development (SDLC, Scrum, Kanban)
• You have experience building and optimizing ‘big data’ data pipelines, architectures and data sets. You have strong analytical skills related to working with both structured and unstructured datasets. You have built processes supporting data transformation, data structures, metadata, dependency and workload management. Strong project management and interpersonal skills. Experience supporting and working with cross-functional teams in a dynamic environment

Responsibilities

• Create, maintain, populate and optimize the CodaMetrix data platform and analytics architecture
• Assemble large, complex data sets that meet functional / non-functional business requirements using the Databricks platform
• Develop and manage ETL processes using Spark and Kafka to ingest, clean, and transform data from different sources (databases, APIs, external feeds, etc.) into usable formats for downstream analysis and reporting
• Implement data quality checks and ensure that data is accurate, consistent, and free from errors
• Implement data governance with data privacy and security regulations (e.g., GDPR, HIPAA)
• Identify, design, and implement internal process improvements such as automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc
• Collaborate with software engineers to ensure that data infrastructure is compatible with applications and services that rely on data
• Optimize data processing workflows for speed, efficiency, and scalability
• Work with stakeholders including the Analytics, Machine Learning, Executive and Product teams to assist with data-related technical issues and support their data infrastructure needs
• Ensure that data infrastructure supports real-time and batch data processing
• Work with structured, semi-structured, and unstructured data, managing large volumes of data and ensuring its accessibility
• Review code, provide constructive feedback, and ensure high standards of engineering excellence within the team
• Lead and mentor junior and mid-level data engineers, providing guidance and training on best practices, architecture design, and data pipeline management
• Establish best practices for data engineering and promote their adoption across teams

Preferred Qualifications

• Preferred Knowledge of HIPAA compliance requirements as well as other security/compliance practices such as PII and SOC2 a big plus
• Experience with Streaming workloads and integrating Spark with Apache Kafka
• Experience with consuming or authoring REST and/or SOAP web service APIs
• Familiarity with machine learning concepts or AI applications in the context of data engineering
• You understand what IaC means and have experience with common tools to implement it

Benefits

• Health Insurance: We cover 80% of the cost of medical and dental insurance and offer vision insurance
• Retirement: We offer a 401(k) plan that eligible employees can contribute to one month after their first day
• Flexibility: We have a generous Paid Time Off policy, which is managed but not limited, so you can take the time you need to relax and rejuvenate
• Learning: All new hires complete our 7-week Onboarding Program where they learn about our company and each of our departments through live sessions hosted by a variety of our leaders
• Development: We provide annual performance evaluations and prioritize working with employees on what their individual growth looks like
• Recognition: We recognize the outstanding achievements of our team through annual company awards where employees have the opportunity to nominate their peers
• Office Location: A modern open plan workspace located in the bustling Back Bay neighborhood of Boston
• Additional Employer Paid Benefits: We offer employer-paid life insurance and short-term and long-term disability insurance