Remote Principal Engineer, Data Pipeline

Logo of Aviatrix

Aviatrix

πŸ’΅ $210k-$280k
πŸ“Remote - United States

Job highlights

Summary

Join our dynamic Data Pipeline team as a Principal Engineer to develop and maintain infrastructure for collecting telemetry at scale by building highly performant real-time data pipelines.

Requirements

  • 10+ years of software development in building large-scale Data Analytics solutions
  • Proficiency in Golang (preferred) or Python
  • Proficiency with common telemetry frameworks (Prometheus, Open Telemetry)
  • Proficiency with big data pipeline technologies (g., Logstash, FluentD, Kafka, Spark, Hadoop, Storm, Spark, Flink)
  • Domain knowledge in networking: (OSI stack, BGP, IP, TCP, UDP, HTTP, DNS; Linux network stack, SDN, network automation)
  • BS in Computer Science, Software Engineering, Machine Learning, Mathematics, Statistics, or equivalent

Responsibilities

  • Design and architect the data pipeline infrastructure needed for network analytics including data collection, storage, and processing components to ensure scalability, reliability, and performance of the pipeline
  • Lead and coordinate with other engineers to identify and collect relevant data from network devices, such as routers, switches, and firewalls
  • Develop algorithms, processes to transform and cleanse the raw network data. This may involve parsing, normalizing, aggregating, and enriching the data to make it suitable for analysis
  • Integrate the data pipeline with various analytics tools and platforms such as data visualization tools and ML
  • Optimize the data pipeline to handle large volumes of network data efficiently, minimizing latency and maximizing throughput. This may involve parallel processing, distributed computing and leveraging cloud
  • Implement data quality checks and validation mechanisms to ensure the accuracy, consistency and completeness of the network data flowing through the pipeline
  • Set up monitoring tools and techniques to monitor the health and performance of the data pipeline

Benefits

  • 100% of employee premiums and 88% of dependent(s) premiums for medical, dental and vision coverage
  • 401(k) match
  • Short and long-term disability
  • Life/AD&D insurance
  • $1,000/year education reimbursement
  • Flexible vacation policy

Share this job:

Disclaimer: Please check that the job is real before you apply. Applying might take you to another website that we don't own. Please be aware that any actions taken during the application process are solely your responsibility, and we bear no responsibility for any outcomes.

Similar Remote Jobs

Please let Aviatrix know you found this job on JobsCollider. Thanks! πŸ™