Build your own search engine with Text Transformers and Qdrant Vector Database
This project has been created to demonstrate how we can build modern Search Engines using a straightforward structure of Text Embeddings (Huggingface Transformers) and a Vector Database.
As a basis for the search data, I used CommonCrawl’s last six months of crawled website data, but because it took almost two weeks on my laptop (M1 Max) to Vectorize ~1mln domains, I assume anyone who will use this repository would not try to vectorize entire dataset 🤷♂️
https://github.com/tigranbs/search-en...
-----
Website: https://tigran.tech
X: https://x.com/tigranbs
Medium: / tigranbs
Github: https://github.com/tigranbs