How To Schedule Scrapy Spiders With Scrapyd & ScrapeOps
In this video we go through how to:
Setup Scrapy & Scrapyd on a brand new Ubuntu server
Clone an example demo project from GitHub
Connect the server to ScrapeOps
Schedule a job
Add the ScrapeOps monitoring SDK
View the data from a successful crawl in ScrapeOps
Link to the GitHub Repo we use in this video: https://github.com/scrapeops/scrapeop...
Link to ScrapeOps: https://scrapeops.io
Get a Digital Ocean server - $100 of free credit by signing up here: https://m.do.co/c/2656441c8345
0:00 - Setting up Scrapy & Scrapyd
4:41 - Adding the server to ScrapeOps
6:35 - Scheduling a job with ScrapeOps
7:43 - Installing the ScrapeOps monitoring and viewing the stats