Effective Web Scraping with Stammer.ai
Sometimes when scraping a website fully you are extracting all good and bad data alike.
These are typically extra links and copy from the header and footer being added that is not relevant, nor provides any context to what a user might ask or want to have a conversation about.
Instead you can "clean" the data prior to be able to have better responses with more accurate information.