Use wget to download / scrape a full website

Опубликовано: 19 Октябрь 2024
на канале: Melvin L

52,380

359

********************************************
Using wget to download/scrape an entire site
********************************************

Resources: https://www.gnu.org/software/wget/man...

Example 1: Basic Example
wget https://docs.scrapy.org/en/latest/

Example 2: Basic Template

wget \
--recursive \
--no-clobber \
--page-requisites \
--html-extension \
--convert-links \
--restrict-file-names=unix \
--domains docs.scrapy.org \
--no-parent \
https://docs.scrapy.org/en/latest/

Example 3: More Advanced Template
wget \
--recursive \
--no-clobber \
--page-requisites \
--html-extension \
--convert-links \
--restrict-file-names=unix \
--domains docs.scrapy.org \
--wait=5 \
--limit-rate=20K \
--user-agent=Mozilla \
--level=10 \
--output-file /home/melvin/test/activity.txt \
--no-parent \
https://docs.scrapy.org/en/latest/ &

Other options
--random-wait