--crawl-delay

CLI

--crawl-delay

REST

crawl_delay

Requires page loads on one host wait at least this many seconds, across every browser of a link crawl or sitemap walk. 0 loads pages without any delay.

A host whose robots.txt asks for a longer Crawl-delay is paced by that instead, with a warning. With an Enterprise license, a --crawl-delay is followed even where robots.txt asks for longer.

Example

zshot -f house-at-pooh-corner.wacz \
  --crawl-delay 3 \
  https://zshot-cli.com/example_assets/house-at-pooh-corner/sitemap.xml

{
  "url": "https://zshot-cli.com/example_assets/house-at-pooh-corner/sitemap.xml",
  "output_type": "wacz",
  "crawl_delay": 3
}

curl -X POST http://127.0.0.1:3000/ \
  -H "Content-Type: application/json" \
  -d '{"url":"https://zshot-cli.com/example_assets/house-at-pooh-corner/sitemap.xml","output_type":"wacz","crawl_delay":3}' \
  -o house-at-pooh-corner.wacz