Data Pipeline Operations Engineer

Summary

Join SixMap, a leader in Automated Cyber Defense Platforms, as a detail-oriented Data Pipeline Operations Engineer. You will manage our weekly scanning process, ensuring timely customer data flow through research, scanning, and UI ingest. This critical role requires a blend of programming, database, and Linux system administration skills. You will monitor and troubleshoot jobs, optimize runtimes, perform data ingest, and execute quality checks. The ideal candidate possesses strong problem-solving and communication skills, along with flexibility to handle off-hours work as needed. SixMap offers a competitive compensation package with excellent benefits.

Requirements

• Strong Linux command line skills
• Experience with Airflow or similar workflow orchestration tools
• Python programming proficiency
• Advanced SQL knowledge for data ingest, refresh, and validation
• Ability to diagnose and resolve issues with long-running batch processes
• Excellent attention to detail and problem-solving skills
• Good communication to coordinate with other teams
• Flexibility to handle off-hours work when needed to meet SLAs

Responsibilities

• Manage the weekly scanning process, ensuring customer data progresses through research, scanning, and UI ingest phases according to defined SLAs
• Prepare input files and kick off processes on the scanning cluster via Airflow
• Monitor and troubleshoot jobs, adjusting parameters like rate files as needed to optimize runtimes
• Perform data ingest into production databases using SQL and Python
• Clear data artifacts and caches in between ingest cycles
• Execute post-ingest data refresh routines
• Perform quality checks on ingested data to validate contractual obligations are met
• Identify process bottlenecks and suggest or implement improvements to the automated tooling to increase speed and reliability

Preferred Qualifications

• Familiarity with network scanning tools and methodologies
• Experience optimizing database performance
• Scripting skills to automate routine tasks
• Understanding of common network protocols and services
• Knowledge of AWS services like EC2

Benefits

• Competitive compensation packages; including equity
• Employer paid medical, dental, vision, disability & life insurance
• 401(k) plans
• Flexible Spending Accounts (health & dependents)
• Unlimited PTO
• Remote Working Options