Senior Search Engineer

Summary

Join Acryl Data, the company behind the leading open-source metadata platform DataHub, as a Search Engineer. You will optimize and scale our Elasticsearch-based search infrastructure, ensuring lightning-fast and highly relevant search experiences across massive datasets. This role demands deep technical expertise in search systems and a data-driven approach to relevance tuning. You will architect and optimize Elasticsearch clusters, implement caching and indexing strategies, develop ranking algorithms, and build robust monitoring and alerting systems. The ideal candidate possesses extensive experience with Elasticsearch, strong coding skills, and a proven track record in search performance optimization. Acryl Data offers a remote-first work environment and a competitive benefits package.

Requirements

• 7+ years of software engineering experience, with at least 4 years focused on search systems
• Deep expertise with Elasticsearch, including: Cluster administration and optimization
• Deep expertise with Elasticsearch, including: Query and indexing performance tuning
• Deep expertise with Elasticsearch, including: Relevance scoring and ranking
• Deep expertise with Elasticsearch, including: Text analysis and processing
• Deep expertise with Elasticsearch, including: Monitoring and observability
• Strong experience with relevance engineering and information retrieval concepts
• Proficiency in performance analysis and optimization techniques
• Experience with large-scale data processing and distributed systems
• Strong coding skills in Java, Python, or similar languages
• Experience with search metrics (NDCG, MRR, etc.) and A/B testing

Responsibilities

• Architect and optimize Elasticsearch clusters for high-throughput, low-latency operations
• Implement and tune caching strategies to optimize query performance
• Design and implement efficient indexing strategies for large-scale data ingestion
• Monitor and optimize cluster health, performance metrics, and resource utilization
• Implement horizontal scaling solutions while maintaining consistent performance
• Design and implement efficient replication and sharding strategies
• Develop and tune ranking algorithms to improve search relevance
• Implement advanced text analysis pipelines using analyzers, tokenizers, and filters
• Design and implement custom scoring functions and boosting strategies
• Create and maintain relevance testing frameworks
• Conduct A/B tests to measure and improve search quality
• Implement semantic search capabilities using vector search and relevant plugins
• Design and implement efficient search APIs and query DSL patterns
• Build robust monitoring and alerting for search infrastructure
• Develop tools for search quality analysis and debugging
• Implement advanced features like faceted search, autocomplete, and filtering
• Create documentation and best practices for search implementation

Preferred Qualifications

• Experience with other search engines (Solr, Lucene) and vector databases
• Knowledge of machine learning for search ranking
• Experience with semantic search and embedding models
• Background in natural language processing
• Experience with search infrastructure in cloud environments (AWS, GCP)
• Contributions to Elasticsearch or related open-source projects

Benefits

• Competitive salary
• Equity
• 99% coverage of medical, dental and vision insurance for US Employees
• Carrot Fertility and Family Planning
• Remote friendly
• Monthly co-working salary
• One-time home office budget