Crawlbot Basics - Choosing The Right Web Data Extraction API For Crawling
In this second Crawlbot basics tutorial, we look at what web data extraction APIs you can pair with Crawlbot. Crawlbot is a web spider that can quickly follow links throughout a site. But you have to tell Crawlbot what to do with the information it gets on each page. Diffbot's Automatic Extraction APIs provide AI-enabled rule-less web extraction for many common page types. For unusual cases, you may need to utilize the Custom API.