Remote Cloud Consultant

Summary

Join our team as a Cloud Consultant and work with globally dispersed cloud specialists to design and create impactful software powering enterprise data platform solutions.

Requirements

• Proficiency in a programming language such as Python, Java, Go or Scala
• Experience with big data cloud technologies like EMR, Athena, Glue, Big Query, Dataproc, Dataflow
• Ideally you will have specific strong hands on experience working with Google Cloud Platform data technologies - Google BigQuery, Google DataFlow, and Executing PySpark and SparkSQL code at Dataproc
• Understand the fundamentals of Spark (PySpark or SparkSQL) including using the Dataframe Application Programming Interface as well as analyzing and performance tuning Spark queries
• Have experience developing and supporting robust, automated and reliable data pipelines
• Develop frameworks and solutions that enable us to acquire, process, monitor and extract value from large dataset
• Have strong SQL skills
• Bring a good knowledge of popular database and data warehouse technologies & concepts from Google, Amazon or Microsoft (Cloud & Conventional RDBMS), such as BigQuery, Redshift, Microsoft Azure SQL Data Warehouse, Snowflake etc
• Have strong knowledge of a Data Orchestration solutions like Airflow, Oozie, Luigi or Talend
• Have strong knowledge of DBT (Data Build Tool) or DataForm
• Experience with Apache Iceberg, Hudi and Query engines like Presto (Trino) is a plus
• Knowledge of Data Catalogs (AWS Glue, Google DataPlex etc.), Data Governance and Data Quality Solutions (for eg. Great Expectations)
• Have knowledge of how to design distributed systems and the trade-offs involved
• Experience with working with software engineering best practices for development, including source control systems, automated deployment pipelines like Jenkins and devops tools like Terraform
• Experience in data modeling, data design and persistence (e.g. warehousing, data marts, data lakes)
• Experience in performing DevOps activities such as IaC using Terraform, provisioning infrastructure in GCP/aws/Azure, defining Data Security layers etc

Responsibilities

• Design and development of end to end Cloud based solutions
• Translate complex functional and technical requirements into detailed designs
• Write high-performance, reliable and maintainable code
• Develop test automation and associated tooling needed for the project
• Work on complex and varied Cloud based projects including tasks such as collecting, parsing, managing, analyzing, and visualizing very large datasets etc
• Maintain and execute DataOps tasks such as performance optimization of ETL/ELT pipeline
• Perform Data Pipeline specific DevOps activities such as Infrastructure provisioning, writing IaC code
• Analyze potential issues and complete root cause analysis and assign issues to be resolved
• Follow up with Data Engineering team members to see fixes through completion
• Review bug descriptions, functional requirements and design documents, incorporating this information into test plans and test cases
• Performance tuning for batch and real-time data processing
• Secure components of clients’ Cloud Data platforms
• Health-checks and configuration reviews
• Data pipelines development – ingestion, transformation, cleansing
• Data flow integration with external systems
• Integration with data access tools and products
• Foundational CI/CD for all infrastructure components, data pipelines, and custom data apps
• Common operational visibility of the data platform from data platform infrastructure to data pipelines, machine learning apps
• Assist client application developers and advise on efficient data access and manipulations
• Define and implement efficient operational processes

Benefits

• Competitive total rewards package with excellent take home salaries
• Shifted work time bonus (if applicable)
• Annual bonus plan!
• Hone your skills or learn new ones with an annual training allowance
• 2 paid professional development days, attend conferences, become certified, whatever you like!
• 3 weeks of paid time off and flexible working hours