Senior Data Engineer - DBA

Summary

Join Oportun as a Sr. Data Engineer and become a key member of our team, designing, developing, and maintaining sophisticated software/data platforms. You will leverage your technical expertise to solve business problems, contribute to architectural decisions, mentor junior engineers, and collaborate with cross-functional teams. This role offers the opportunity to lead technology efforts for large initiatives, from requirements gathering to final product delivery. You will be responsible for database design and architecture, performance monitoring and tuning, security and compliance, backup and recovery, data integration and ETL support, and incident response. The position requires expertise in various database technologies, big data technologies, and ETL processes. You will also collaborate with developers, data scientists, and DevOps engineers.

Requirements

• Bachelor's or master’s degree in computer science, Data Science, or a related field
• 5+ years of experience in data engineering, with a focus on data architecture, ETL, and database management
• Proficiency in programming languages like Python/PySpark and Java or /Scala
• Expertise in big data technologies such as Hadoop, Spark, Kafka, etc
• In-depth knowledge of SQL and experience with various database technologies (e.g., PostgreSQL, MariaDB MySQL, NoSQL databases)
• Experience and expertise in building complex end-to-end data pipelines
• Experience with orchestration and designing job schedules using the CICD tools like Jenkins, Airflow or Databricks
• Ability to lead ETL migration from Talend to Databricks pyspark
• Demonstrated building capabilities, reusable utilities, and tools for speeding complex business processes
• Ability to work in an Agile environment (Scrum, Lean, Kanban, etc)
• Ability to mentor junior team members
• Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and their data services (e.g., AWS Redshift, S3, Azure SQL Data Warehouse)
• Strong leadership, problem-solving, and decision-making skills
• Excellent communication and collaboration abilities

Responsibilities

• Database Design & Architecture Design, implement, and maintain optimal database schemas for relational (MariaDB) and NoSQL (MongoDB) databases
• Participate in data modeling efforts to support analytics in Databricks
• Performance Monitoring & Tuning Monitor and tune performance across all platforms to ensure optimal performance
• Use profiling tools (e.g., EXPLAIN, query plans, system logs) to identify and resolve bottlenecks
• Security & Compliance Implement access controls, encryption, and database hardening techniques
• Manage user roles and privileges across MariaDB, MongoDB, and Databricks
• Ensure compliance with data governance policies (e.g., GDPR, HIPAA)
• Backup & Recovery Implement and maintain backup/recovery solutions for all database platforms
• Periodically test restore procedures for business continuity
• Data Integration & ETL Support Support and optimize ETL pipelines between MongoDB, MariaDB, and Databricks
• Work with data engineers to integrate data sources for analytics
• Monitoring & Incident Response Set up and monitor database alerts
• Troubleshoot incidents, resolve outages, and perform root cause analysis
• MariaDB-Specific Responsibilities Administer MariaDB instances (standalone, replication, Galera Cluster)
• Optimize SQL queries and indexing strategies
• Maintain stored procedures, functions, and triggers
• Manage schema migrations and upgrades with minimal downtime
• Ensure ACID compliance and transaction management
• MongoDB-Specific Responsibilities Manage replica sets and sharded clusters
• Perform capacity planning for large document collections
• Tune document models and access patterns for performance
• Set up and monitor MongoDB Ops Manager / Atlas (if used)
• Automate backup and archival strategies for NoSQL data
• Databricks-Specific Responsibilities Manage Databricks workspace permissions and clusters
• Collaborate with data engineers to optimize Spark jobs and Delta Lake usage
• Ensure proper data ingestion, storage, and transformation in Databricks
• Support CI/CD deployment of notebooks and jobs
• Integrate Databricks with external data sources (MariaDB, MongoDB, S3, ADLS)
• Collaboration & Documentation Collaborate with developers, data scientists, and DevOps engineers
• Maintain up-to-date documentation on data architecture, procedures, and standards
• Provide training or onboarding support for other teams on database tools

Preferred Qualifications

• Familiarity or certification in Databricks is a plus
• MariaDB Tools: mysqldump, mysqladmin, Percona Toolkit
• MongoDB Tools: mongodump, mongotop, mongoexport, Atlas UI
• Databricks Tools: Jobs UI, Databricks CLI, REST API, SQL Analytics
• Scripting: Bash, Python, PowerShell
• Monitoring: Prometheus, Grafana, CloudWatch, DataDog
• Version Control & CI/CD: Git, Jenkins, Terraform (for infrastructure-as-code)
• Preferred Cloud provider: AWS