Python Scrapy - Scraping LinkedIn Company Profiles
In this video, we look at how you can scrape company profiles from LinkedIn using Python Scrapy. We look at how to use Scrapy and a proxy to request the profile for a specific company and then parse the associated profile data, education, job experience etc.
The article that goes along with this video:
https://scrapeops.io/python-scrapy-pl...
Basic Scrapy project used in this tutorial:
https://github.com/python-scrapy-play...
Completed Scrapy Project which you can just download and run (after adding your account details!!): https://github.com/python-scrapy-play...
Link to the ScrapeOps proxy: https://scrapeops.io/proxy-aggregator/
The proxy python library: https://pypi.org/project/scrapeops-sc...
00:00 - Intro
01:02 - Understanding how LinkedIn & company pages are structured
03:45 - Setting up a basic spider
05:33 - Setting up a proxy to by-pass LinkedIn restrictions (anti-bots)
07:30 - Writing our spider code
21:38 - Running our spider
24:13 - Reading multiple company URLs from a file
28:11 - Running our updated spider
30:08 - Outro