Senior Data Engineer

Summary

Join Thoughtworks as a Senior Data Engineer and build, maintain, and test software architecture and infrastructure for data applications. You will develop core capabilities for technical and functional data platforms, acting as an anchor for work streams and ensuring timely delivery. Leveraging the latest big data tools and frameworks, you'll collaborate on problem-solving and strategy execution. You'll design and operate modern data architectures, develop intricate data processing pipelines, and collaborate with data scientists. The role involves writing clean, iterative code, utilizing various technologies, and contributing to data governance, security, and privacy. Thoughtworks offers a supportive and collaborative environment with opportunities for professional development.

Requirements

• Deep understanding of Databricks platform, including its architecture, features, and best practices
• Proficiency in using Databricks workspaces, notebooks, clusters, and jobs for data processing and analysis
• Experience with Databricks SQL, Delta Lake, and other Databricks-specific technologies
• Solid knowledge of AWS cloud services relevant to data engineering, such as S3, EC2, EMR, Glue, and Redshift
• Ability to design, deploy, and manage data pipelines and infrastructure on AWS using Databricks
• Strong grasp of big data processing concepts and techniques, including ETL, data warehousing, and data lake architectures
• Expertise in using Apache Spark for large-scale data processing and transformation within Databricks
• Proficiency in programming languages commonly used in data engineering, such as Python or SQL
• Ability to write efficient and maintainable code for data processing, automation, and integration tasks
• Solid understanding of data modeling, data governance, and data quality principles and knowledge of data pipeline development, deployment, and monitoring using CI/CD tools
• You understand the importance of stakeholder management and can easily liaise between clients and other key stakeholders throughout projects, ensuring buy-in and gaining trust along the way
• You are resilient in ambiguous situations and can adapt your role to approach challenges from multiple perspectives
• You don’t shy away from risks or conflicts, instead you take them on and skillfully manage them
• You are eager to coach, mentor and motivate others and you aspire to influence teammates to take positive action and accountability for their work
• You enjoy influencing others and always advocate for technical excellence while being open to change when needed

Responsibilities

• Develop and operate modern data architecture approaches to meet key business objectives and provide end-to-end data solutions
• Develop intricate data processing pipelines, addressing clients' most challenging problems
• Collaborate with data scientists to design scalable implementations of their models
• Write clean, iterative code using TDD and leverage various continuous delivery practices to deploy, support and operate data pipelines
• Use different distributed storage and computing technologies from the plethora of options available
• Develop data models by selecting from a variety of modeling techniques and implementing the chosen data model using the appropriate technology stack
• Collaborate with the team on the areas of data governance, data security and data privacy
• Incorporate data quality into your day-to-day work

Benefits

• There is no one-size-fits-all career path at Thoughtworks: however you want to develop your career is entirely up to you. But we also balance autonomy with the strength of our cultivation culture. This means your career is supported by interactive tools, numerous development programs and teammates who want to help you grow. We see value in helping each other be our best and that extends to empowering our employees in their career journeys
• #LI-Remote