Senior Data Engineer

Summary

Join Oportun as a Sr. Data Engineer and become a key member of our team, designing, developing, and maintaining sophisticated data platforms. You will lead the design and implementation of scalable data architectures, develop data pipelines, manage databases, and ensure data quality. Your expertise will be crucial in solving business problems with technical solutions, mentoring junior engineers, and collaborating with cross-functional teams. This role offers the opportunity to lead large, multi-month projects from requirements gathering to final delivery. You will contribute to architectural decisions and ensure high-quality, scalable software solutions.

Requirements

• Have a strong understanding of a business or system domain with sufficient knowledge & expertise around the appropriate metrics and trends
• Provide technical leadership and expertise, guiding the team in making sound architectural decisions and solving challenging technical problems
• Conduct code reviews and provide constructive feedback to ensure code quality, performance, and maintainability
• Mentor and coach junior engineers, fostering a culture of continuous learning, growth, and technical excellence within the team
• Play a significant role in the ongoing evolution and refinement of current tools and applications used by the team, and drive adoption of new practices within your team
• Take ownership of (customer) issues, including initial troubleshooting, identification of root cause and issue escalation or resolution, while maintaining the overall reliability and performance of our systems
• Set the benchmark for responsiveness and ownership and overall accountability of engineering systems
• Independently drive and lead multiple features, contribute to (a) large project(s) and lead smaller projects
• Orchestrate work that spans multiples engineers within your team and keep all relevant stakeholders informed
• Support your lead/EM about your work and that of the team, that they need to share with the stakeholders, including escalation of issues
• Bachelor's or Master's degree in Computer Science, Data Science, or a related field
• 5+ years of experience in data engineering, with a focus on data architecture, ETL, and database management
• Proficiency in programming languages like Python/PySpark and Java or Scala
• Expertise in big data technologies such as Hadoop, Spark, Kafka, etc
• In-depth knowledge of SQL and experience with various database technologies (e.g., PostgreSQL, MariaDB, NoSQL databases)
• Experience and expertise in building complex end-to-end data pipelines
• Experience with orchestration and designing job schedules using the CICD tools like Jenkins, Airflow or Databricks
• Ability to work in an Agile environment (Scrum, Lean, Kanban, etc)
• Ability to mentor junior team members
• Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and their data services (e.g., AWS Redshift, S3, Azure SQL Data Warehouse)
• Strong leadership, problem-solving, and decision-making skills
• Excellent communication and collaboration abilities

Responsibilities

• Lead the design and implementation of scalable, efficient, and robust data architectures to meet business needs and analytical requirements
• Collaborate with stakeholders to understand data requirements, build subject matter expertise, and define optimal data models and structures
• Design and develop data pipelines, ETL processes, and data integration solutions for ingesting, processing, and transforming large volumes of structured and unstructured data
• Optimize data pipelines for performance, reliability, and scalability
• Oversee the management and maintenance of databases, data warehouses, and data lakes to ensure high performance, data integrity, and security
• Implement and manage ETL processes for efficient data loading and retrieval
• Establish and enforce data quality standards, validation rules, and data governance practices to ensure data accuracy, consistency, and compliance with regulations
• Drive initiatives to improve data quality and documentation of data assets
• Provide technical leadership and mentorship to junior team members, assisting in their skill development and growth
• Lead and participate in code reviews, ensuring best practices and high-quality code
• Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand their data needs and deliver solutions that meet those needs
• Communicate effectively with non-technical stakeholders to translate technical concepts into actionable insights and business value
• Implement monitoring systems and practices to track data pipeline performance, identify bottlenecks, and optimize for improved efficiency and scalability

Preferred Qualifications

Familiarity or certification in Databricks is a plus