Senior Staff Security Detection Engineer

Summary

Join Databricks' Detection & Response team as a Security Detection Engineer and play a critical role in protecting our products and infrastructure from cyber threats. You will design and implement scalable intrusion detection solutions using machine learning and cutting-edge technologies. Collaborate with data engineering and incident response teams to optimize log ingestion, develop detection models, and perform threat hunting. This individual contributor role requires extensive experience in security detection engineering, software engineering, and cloud security. You will leverage your expertise in Python, Spark, and other technologies to build and deploy detection solutions on the Databricks platform. Your work will directly impact Databricks' security infrastructure and contribute to the evolution of our detection systems. The role is remote and offers a competitive compensation package.

Requirements

• 12+ years of relevant experience or advanced degree + 8 years of experience, with a focus on security detection engineering
• 8+ years of software engineering experience, with 5+ years specifically in security-related engineering, particularly in detection engineering
• Expertise in securing and operating within multiple cloud environments (AWS, Azure, GCP)
• Strong technical proficiency in key areas such as network security, cloud security, application/log analysis, and endpoint security
• Proven experience in Python, Git/GitHub, and CI/CD automation (terraform knowledge is a plus)
• Familiarity with distributed computing environments (e.g., Pyspark), SQL, and data analysis tools
• Demonstrated ability to apply machine learning techniques to security problems
• A strong passion for continuous learning and staying updated on evolving attack techniques and defense strategies
• Excellent communication skills, with the ability to collaborate effectively across teams and present complex ideas clearly to stakeholders at all levels
• A leadership mindset with the ability to mentor peers, drive strategic initiatives, and influence the organization’s security direction

Responsibilities

• Design and implement advanced detection strategies by deeply understanding and analyzing new or unknown log sources, schemas, and raw data
• Collaborate with cross-functional teams, including data engineering, to build efficient log ingestion pipelines and support large-scale data analytics
• Engineer and deploy detection solutions on Databricks using Spark, Python, and other cutting-edge technologies with a strong emphasis on clean code, rigorous testing, and comprehensive documentation
• Develop Rule-based and/or ML-based intrusion detection models and integrate them with Databricks' platform, ensuring high accuracy and minimal false positives
• Partner with Incident Response teams to perform threat hunting and to provide detailed logging, alerts, and playbooks, empowering proactive threat detection and response
• Influence the development of long-term technology strategies and roadmaps for detection engineering, ensuring alignment with broader business and security goals
• Represent Databricks at security and engineering conferences, presenting novel detection approaches and thought leadership within the security community

Preferred Qualifications

Terraform knowledge

Benefits

• Annual performance bonus
• Equity
• Comprehensive benefits and perks