Scrapy Redis: Scaling Your Scrapy Spiders Using Distributed Workers
In this video, we look at how you can scale out and speed up your web scraping using multiple workers(Spiders) using scrapy-redis.
The article that goes along with this video:
https://scrapeops.io/python-scrapy-pl...
Scrapy project used in this tutorial:
https://github.com/python-scrapy-play...
Where you can sign up for a free cloud Redis instance:
https://redis.com/try-free/
Where you can download RedisInsite the Redis desktop manager:
https://redis.com/redis-enterprise/re...
00:00 - Intro
00:37 - Things to think about when scaling
03:01 - Setting up Redis
05:31 - Adding list of urls to be scraped to Redis queue
08:21 - Scraping with one worker (spider)
12:49 - Scraping with two workers (spiders)
16:59 - Outro