Senior Big Data Engineer

Summary

Join our team as a Senior Big Data Engineer and leverage your Scala and Apache Spark expertise to support our data engineering team. You will analyze and refactor existing code, develop enhancements, create robust unit tests, and help migrate code to a modern cloud-native environment. This role requires proficiency in Scala and Apache Spark, experience with AWS services (S3, Glue, Step Functions), and working knowledge of MongoDB and Python. You will collaborate within an Agile team, contributing to sprint planning and code reviews. Strong problem-solving skills, attention to detail, and excellent communication are essential. The position offers the flexibility of working remotely, hybrid, or in-office locations (CLT/NY/NJ).

Requirements

• Proficiency in Scala and Apache Spark for large-scale data processing
• Working knowledge of Python for scripting and data transformation
• Hands-on experience with Amazon Web Services (AWS), including: S3, AWS Glue, Step Functions
• Experience working with MongoDB and structured/unstructured data
• Familiarity with SQL and BDD frameworks
• Strong understanding of software development lifecycle (SDLC), from development to production deployment
• Experience using tools like Git, Azure DevOps/TFS, and JIRA
• Comfortable working in Agile/Scrum environments

Responsibilities

• Analyze and refactor existing Scala Spark code to support migration to new environments
• Write and enhance Scala Spark code with a focus on performance and maintainability
• Develop comprehensive unit tests to ensure code quality and stability
• Work with data stored in Amazon S3 (CSV and Parquet formats)
• Integrate and manipulate data from MongoDB
• Independently set up environments to validate and test code changes
• Execute both manual and automated testing procedures
• Collaborate within an Agile team and contribute to sprint planning, story grooming, and reviews
• Maintain clear and proactive communication with team members, project managers, and stakeholders

Preferred Qualifications

• Experience with Terraform for infrastructure-as-code is a plus
• Strong problem-solving abilities and attention to detail
• Ability to work independently and as part of a collaborative Agile team
• Excellent written and verbal communication skills