Crawling

--crawl-links archives a site by following its links. A sitemap target archives the pages it lists, as --crawl-url-is-sitemap controls. Both write WARC or WACZ.

Pacing

Page loads on one host wait 1 second apart. --crawl-delay sets the wait, and 0 removes it. A host whose robots.txt asks for a longer Crawl-delay is paced by that instead; --skip-robots-txt ignores it.

A crawl runs one browser per CPU, and --crawl-concurrency sets the count. The delay holds across every browser, so more browsers speed a crawl only across several hosts or with a shorter delay.

Respect Site Ownership

ZShot follows robots.txt by default, its pacing included. Crawl at a rate the site’s owner would accept.