Doing SEO research with Screaming Frog
This is what it's like doing SEO research with Screaming Frog.
This company I’m investigating ranks for 24,000 keywords.
I'm going to try crawling it to see all of its pages.
But then… it doesn’t work. It just returned one URL back.
I checked its robots.txt file, my crawler is blocked.
I tried to switching to Googlebot.
It still didn't work.
This could be because my crawler doesn't have a valid Googlebot IP address, or another reason.
I tried switching User Agent to Microsoft Edge.
Bingo.
It worked.
I wanted to see how many clicks from the homepage it takes to get to a bunch of pages. As well as site structure and page titles and meta descriptions.
Crawling a site and seeing all of its data makes it easy.
#seo #searchengineoptimization #digitalmarketing