ElasticSearch in Python #36 - Hybrid search with Reciprocal Rank Fusion
Hi everyone! In this video, I'll show you how to perform hybrid search with Elasticsearch and Python by combining semantic (kNN) and full-text search results using a library called ranx.
Hybrid search leverages the strengths of both semantic search, which understands the meaning and context of a query, and traditional full-text search, which excels at keyword matching.
We'll start by explaining the concept and the main challenge: how to effectively combine the results from these two different search methods, especially when their relevance scores are on completely different scales.
I'll introduce the solution, a fusion algorithm called Reciprocal Rank Fusion (RRF). We'll discuss how RRF works to merge ranked lists from multiple sources into a single, more relevant list.
While Elasticsearch offers a built-in RRF feature, it requires a paid Enterprise license. To work around this, I'll demonstrate a free and open-source solution using the ranx Python library.
You'll learn how to set up your Elasticsearch index, embed your documents using a sentence-transformer model, run kNN and full-text queries independently, and then use ranx to fuse the results into a unified ranking.
Here is the link to the GitHub repository for the course: https://github.com/ImadSaddik/Elastic...
Useful links:
ranx library: https://github.com/AmenRa/ranx
Elasticsearch documentation on RRF: https://www.elastic.co/docs/reference...
The embedding model: https://huggingface.co/sentence-trans...
Don't forget to like, subscribe, and leave a comment if you have any questions or feedback!
⭐️ Contents ⭐️
(00:00) Intro to Hybrid Search
(01:25) Combining results with Reciprocal Rank Fusion (RRF)
(02:07) RRF licensing requirements in Elasticsearch
(02:43) Using the 'ranx' library as a free alternative
(03:54) Connecting to Elasticsearch with the Python client
(04:23) Preparing the index for vector search
(04:45) Loading the sentence-transformer embedding model
(05:33) Loading and embedding the documents
(07:07) Performing a k-Nearest Neighbor (kNN) search
(08:11) Performing a full-text search
(09:52) Paid Solution: How to use the built-in RRF feature
(11:02) Free Solution: Fusing results with the ranx library
(12:43) Implementing the RRF fusion logic in Python
(14:47) Reviewing the final combined search results
(16:30) Outro
#Elasticsearch #Python #HybridSearch #SemanticSearch #VectorSearch #RRF #ReciprocalRankFusion #DataScience #ranx