Remote Principal Data Engineer

Summary

Join our fast-paced, dynamic team of experts as a Principal Data Engineer and contribute to the development of scalable enterprise platforms in AWS. As a Principal Data Engineer, you will champion best practices and a well-architected approach to implementing cloud data solutions, covering industry-leading open source and cloud services such as observability, dataOps, data application deployments, Spark, Python, and other application and platform management technologies.

Requirements

• 8-12+ years working in data engineering, including big data, ETL, Spark/PySpark, data warehousing, and analytics application development. Demonstrated familiarity with underlying technology architectures rather than vendor solutions
• 8+ years of experience with query languages and database technologies
• 5+ years of experience working with cloud computing platforms such as AWS
• 5+ years of experience with Python and the Python ecosystem (PySpark, Pandas, DataFrames, etc.), showcasing the ability to write clean, maintainable, and robust code
• Experience ingesting files (XML/CSV) and normalizing data
• Experience in building data pipelines in production and the ability to work across structured, semi-structured, and unstructured data
• Demonstrated experience working with external data vendors and managing data partnerships
• Experience with version control systems, preferably Git
• Ability to work in a Linux environment
• Strong knowledge of data & software engineering concepts and best practices
• Experience in preparing data for analytics and following data workflows
• Knowledge of systems monitoring tools in a cloud environment

Responsibilities

• Designing and deploying medium to big data applications fit for purpose across scaling, throughput, and high-availability dimensions
• A focus on DORA/DevOps delivery best practices including delivering to production frequently with small batch sizes, focusing on learning from clients
• Working with global teams and product owners, providing industry-leading cloud data solutions and applying best practices
• Supporting security policies, standards, and processes based on current and previous business requirements
• Creating and maintaining comprehensive documentation
• Partnering with operations teams to onboard business-as-usual work and automation to the appropriate groups
• Creating and deploying robust and reusable config management solutions

Preferred Qualifications

• The ideal candidate can demonstrate the ability to deliver value to clients that drive positive business outcomes
• Has been a tech lead or technical leader on an empowered engineering team, working closely with a product manager and clients to understand the feasibility of building minimum viable features
• Experience working with Terraform for Infrastructure as Code (IaC) in AWS
• Experience working with LLMs/ML/AI
• Fintech background is a bonus