Data Engineer

Summary

Join Quisitive's Data & AI practice as an Azure Data Engineer, designing, building, and maintaining scalable data pipelines and analytics solutions on the Microsoft Azure platform. You will utilize cutting-edge technologies like Azure Databricks, Azure Synapse Analytics pipelines, and Microsoft Fabric. This role requires expertise in Azure data services and a strong background in data engineering (ETL/ELT). The ideal candidate is proficient in optimizing data systems and collaborating with various teams. The position is remote, anywhere in the US, with a preference for the Atlanta metro area. Quisitive offers a high-performing culture and resources to support your success.

Requirements

• 3+ years of experience in data engineering or BI development, with a focus on the Azure cloud data platform
• Azure Data Services Expertise: Hands-on experience with Azure Databricks, Azure Synapse Analytics (particularly Synapse pipelines and SQL pools), Azure Data Factory, and Azure Data Lake Storage. Familiarity with Microsoft Fabric or eagerness to learn it, as we are adopting Fabric for unified analytics
• Programming & DB Proficiency: Strong coding skills in Python (for data manipulation and scripting) and SQL (for querying and transforming data)
• ETL/ELT and Data Modeling: Proven ability to design and implement ETL/ELT workflows. Solid understanding of relational databases and data modeling concepts (normalization, dimensional modeling)
• Experience building data warehouses or data marts on Azure (SQL Database/Synapse)

Responsibilities

• Design & Develop Data Pipelines: Create end-to-end data pipelines using Azure tools (e.g., Azure Data Factory and Synapse Pipelines) to ingest, transform, and load data from various sources into data lakes, data warehouses, and other storage solutions
• Azure Databricks & Spark Processing: Utilize Azure Databricks (Spark) for big data processing tasks. Develop and optimize PySpark/Scala jobs to cleanse, transform, and aggregate large datasets in both batch and real-time streaming modes
• Data Warehousing & Modeling: Build and maintain data warehouses/lakehouses on Azure Synapse or Microsoft Fabric. Design star-schema or snowflake data models that enable efficient analytics and reporting, implementing best practices for performance and maintainability
• Optimize & Monitor Workloads: Tune queries, pipelines, and Spark jobs for performance and cost-efficiency. Monitor pipeline runs, data storage, and processing metrics to ensure reliability and address issues proactively (set up alerts, logging, and automation for failures/retries)
• Collaborate with Teams: Work closely with data analysts, BI developers, and data scientists to understand data needs. Provide clean, well-structured data sets that meet business requirements. Collaborate with cloud architects to ensure solutions are aligned with Azure best practices
• Implement Data Quality & Security: Ensure data quality through validation checks and testing. Implement data security measures such as encryption, access control (Azure RBAC), and data masking as needed. Maintain data governance standards and documentation for data pipelines and datasets
• Stay Current & Innovate: Continuously research and suggest new ways to improve existing data architecture. Stay up-to-date with the latest Azure data services (like Microsoft Fabric and other emerging technologies) and bring innovative ideas to optimize our data platform

Preferred Qualifications

• Experience developing data processing scripts/notebooks in PySpark or Spark SQL
• Knowledge of Scala or .NET for data engineering
• Previous experience with a Microsoft systems integrator
• Microsoft Fabric Data Engineer Associate Certification