Web Crawling and Semantic Search Engine for Faculty Research
In this project, we created a faculty search engine designed to efficiently retrieve faculty profiles based on user queries. The engine automates web crawling using BeautifulSoup, extracts structured data from HTML content, and preprocesses text using SpaCy for tokenization, stop word removal, and lemmatization.
Key highlights include:
1) Inverted Indexing: Built using TF-IDF vectorization from scikit-learn, enabling efficient query matching.
2) Ranking: Implemented the Vector Space Model and used cosine similarity to rank search results based on relevance.
3) Spell Checking: Incorporated PySpellChecker to handle user typos and suggest corrections, enhancing usability.
4) Database: Leveraged MongoDB to store raw HTML, processed faculty information, and document embeddings.
Future Improvements: Exploring advanced models like BERT and RoBERTa, scaling to larger datasets, and deploying as a web application.
Keywords: Faculty Search Engine, TF-IDF, Cosine Similarity, Vector Space Model, PySpellChecker, BeautifulSoup, SpaCy, MongoDB, NLP, Web Crawling.