Data Engineer

Summary

Join Agile Defense as a Data Engineer and contribute to the development and maintenance of data pipelines and analytics products. You will work in an Agile environment, collaborating with cross-functional teams to integrate data from multiple sources. Responsibilities include designing, building, and maintaining ETL/ELT pipelines, developing SQL queries, implementing data quality checks, and optimizing database performance. The ideal candidate will have 3-5 years of experience in a similar role, proficiency in SQL, PySpark, and Bash scripting, and experience with AWS and other cloud technologies. Agile Defense offers a comprehensive benefits package including health, dental, vision insurance, 401k matching, PTO, paid holidays, and tuition assistance. The position is remote.

Requirements

• 3-5 Years of Experience working in an Agile and Scrum software development environment performing the duties of a Data Engineer
• Databricks notebooks, job schedules, SQL query tools
• Managing multi-hope data processing pipelines (185 separate nodes in data pipeline)
• Scripting in  SQL, pyspark, and bash shell
• Postgres database management
• FlaskAPI
• AWS Cloudflare, S3, EC2, EKS, ALB
• Docker containerization
• Apigee
• Jenkins, GitHub and separate code environments

Responsibilities

• Strong oral and written communication skills
• Able to communicate effectively with Product Owners (POs) and stakeholders to understand requirements/scope and be able to translate that into user stories
• Able to effectively communicate and coordinate between stakeholders, POs, and team members including designers, business analysts, developers, and Scrum Masters across other development teams
• Work to become a subject matter expert for the products that are supported
• Able to assist with testing/communication/de-bugging with external integration partner system teams
• After team onboarding, work independently to continue to learn the system and products by reviewing resources and documentation, testing, and engaging with design, development, Product Owner and stakeholders in great detail
• Design, build, and maintain efficient ETL/ELT data pipelines
• Develop and optimize SQL queries for large-scale data processing
• Implement data quality checks and monitoring systems
• Create and maintain database schemas and data models
• Collaborate with cross-functional teams to integrate data from multiple sources
• Optimize database performance through indexing, query analysis, and partitioning
• Automate data workflows using CI/CD pipelines

Preferred Qualifications

• Familiar with and experience using tools like Jira, Confluence, Mural, and Teams to communicate requirements, design, processes, and content
• Experience working in an Agile software development environment and familiar with Scrum structure and ceremonies
• Knowledge of cloud platforms (AWS, Azure, etc.)
• Experience with Jenkins or similar CI/CD tools
• Familiarity with data visualization tools
• USCIS or DHS Experience

Benefits

• Health/dental/vision insurance coverage
• 401K with company match
• PTO & paid holidays
• Annual tuition/training assistance