Principal Engineer, Data Pipeline

Summary

Join our dynamic Data Pipeline team as a Principal Engineer to develop and maintain infrastructure for collecting telemetry at scale by building highly performant real-time data pipelines.

Requirements

• 10+ years of software development in building large-scale Data Analytics solutions
• Proficiency in Golang (preferred) or Python
• Proficiency with common telemetry frameworks (Prometheus, Open Telemetry)
• Proficiency with big data pipeline technologies (g., Logstash, FluentD, Kafka, Spark, Hadoop, Storm, Spark, Flink)
• Domain knowledge in networking: (OSI stack, BGP, IP, TCP, UDP, HTTP, DNS; Linux network stack, SDN, network automation)
• BS in Computer Science, Software Engineering, Machine Learning, Mathematics, Statistics, or equivalent

Responsibilities

• Design and architect the data pipeline infrastructure needed for network analytics including data collection, storage, and processing components to ensure scalability, reliability, and performance of the pipeline
• Lead and coordinate with other engineers to identify and collect relevant data from network devices, such as routers, switches, and firewalls
• Develop algorithms, processes to transform and cleanse the raw network data. This may involve parsing, normalizing, aggregating, and enriching the data to make it suitable for analysis
• Integrate the data pipeline with various analytics tools and platforms such as data visualization tools and ML
• Optimize the data pipeline to handle large volumes of network data efficiently, minimizing latency and maximizing throughput. This may involve parallel processing, distributed computing and leveraging cloud
• Implement data quality checks and validation mechanisms to ensure the accuracy, consistency and completeness of the network data flowing through the pipeline
• Set up monitoring tools and techniques to monitor the health and performance of the data pipeline

Benefits

• 100% of employee premiums and 88% of dependent(s) premiums for medical, dental and vision coverage
• 401(k) match
• Short and long-term disability
• Life/AD&D insurance
• $1,000/year education reimbursement
• Flexible vacation policy