Recursive Web Crawler with Web Scraping in Python
In this video, we use async io and the requests-HTML library in order to build an efficient and effective web crawler. When we find the 'history' word in the HTML content we keep the URL in a list and later on we save it to the MySQL Database.
You can find the code on the GitHub link:
https://github.com/apostolos1927/crawler
Stay tuned for more Data Engineering videos!
Follow me on social media:
LinkedIn: / apostolos-athanasiou-9a0baa119
GitHub: https://github.com/apostolos1927
00:00 - Intro
00:55 - Checking the sources
03:47 - MySQL table & libraries
14:45 - Building the web crawler
20:22 - Store data into Database