Amazon scraping with MechanicalSoup
Today we'll see how to scrape product information from Amazon. In detail, we'll use the Italian localized version of the Amazon portal (amazon.it). We'll scrape these fields:
url
name
price
number of reviews
global rating
and other fields
The data output of the script is in JSON.
We'll see some tips and tricks on how to get and to normalize the scraped data. There are two versions of the script: the first one is a bottom-up approach while the second one results in a much more efficient version. The first script scrapes the product information within the product page itself so if we have n products it costs o(n) requests (at least n requests), while the second scripts costs o(1) requests for n products.
If your script interacts too much with the website in a short amount of time you'll hit some imposed limits: you'll be temporarily blocked!
Links
blog post with full source code: https://blog.franco.net.eu.org/post/s...
MechanicalSoup (integrates BeautifulSoup & Requests): https://mechanicalsoup.readthedocs.io...
BeautifulSoup: https://www.crummy.com/software/Beaut...
Requests: https://requests.readthedocs.io/en/la...
CHAPTERS
0:00 Intro
0:23 The data we'll use
1:16 Product dataclass
4:01 Other dataclasses
5:06 Scraping values from the product page
20:12 Querying the search bar
22:54 Optimized script
26:09 Outro
#scraping #python #beautifulsoup #programming