Robots.txt & Its Impact on SEO
Robots.txt, or robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.
This tiny text file is part of every website on the Internet, but most people don't even know it exists. It can have a significant impact on SEO, if utilized correctly.
Robots.txt is a file that tells search engine crawlers not to crawl certain pages or sections of a website. Most major search engines (including Google, Bing, and Yahoo) recognize and respect the settings in robots.txt.
This file is primarily used to prevent your site from experiencing excessive request load. Robots.txt is not a mechanism for hiding web pages from Google. To hide a page from Google, you must block indexing with the noindex option or password protect the page.
This video is sourced from a Projasaweb article:
https://projasaweb.com/robot-txt/
Don't forget to visit the Projasaweb Academy: