How to use Python to parse JSON sitemaps | Flatten nested dictionaries to get codes for WEB SCRAPING
If you are web scraping with Scrapy, you may want to scrape many categories, but not just scrape all links with a crawler.
If you can find a sitemap, in JSON format, you can flatten the structure, with its lists and dictionaries and then make a new list to use for your URLS or form query string parameters for your URLs to scrape.
Sound like hard work? Not really, 8 lines of code inside a function and off you go. Just print "type" regularly to check what type you are iterating through...
Timings:
0:00 Intro - About sitemaps
4:05 - Start the code
19:00 - Using slice to get 'code' and 'name'
Any questions, add a comment, I'll be pleased to reply!
Dr Pi.
#webscraping #json #sitemap