Scrapy Selenium - Scraping Javascript Rendered Websites (2022)
In this video, we go through how to scrape data from javascript rendered websites using Scrapy Selenium.
Link to the project on GitHub: https://github.com/python-scrapy-play...
The in-depth article with the code snippets we use in the video: https://scrapeops.io/python-scrapy-pl...
If you have any questions or comments - please let us know!
Chapters:
00:00 - Intro
00:27 - Installing base Scrapy project
01:52 - Installing Chromedriver
03:12 - Updating our Scrapy settings to enable the new middleware
04:30 - Adding the code to our Scrapy Spider
08:53 - Adding in a wait for an element to show on a page
10:22 - Clicking a button using a piece of javascript code in our spider
13:27 - Taking a screenshot with Scrapy Selenium while scraping
14:45 - Outro