Data Engineer

Summary

Join our team as a Data Engineer for a 6-month fully remote contract based in the UK. This role requires Active SC Clearance. You will be working with data pipelines, utilizing Python/PySpark frameworks, and handling sensitive datasets. Strong understanding of data concepts, structures, and security principles is crucial. Experience with various technologies, including Azure Databricks, Apache Spark, and Delta Tables, is essential. Proficiency in JSON, YAML, Python, SQL, PySpark, and other specified languages and frameworks is required.

Requirements

• Possess a strong understanding of data concepts, data types, data structures, schemas (both JSON and Spark), and schema management
• Have a strong understanding of complex JSON manipulation
• Possess a strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each
• Possess a strong understanding of Data Security principles, data owners, access controls, row and column level, GDPR etc
• Proficiency in JSON
• Proficiency in YAML
• Proficiency in Python (as a programming language)
• Proficiency in SQL
• Proficiency in PySpark
• Proficiency in Delta Lake
• Proficiency in Bash (both CLI usage and scripting)
• Proficiency in Git
• Proficiency in Markdown
• Possess Active SC Clearance

Responsibilities

• Work with Data Pipelines using a custom Python/PySpark frameworks
• Manage and handle Reference Data
• Handle sensitive datasets
• Demonstrate strong problem-solving and analytical skills

Preferred Qualifications

• Proficiency in Scala
• Experience with Azure SQL Server as a HIVE Metastore
• Pydantic experience

Benefits

Fully Remote Working