# Crawling

Source: https://zshot-cli.com/reference/crawling/
Markdown: https://zshot-cli.com/reference/crawling/index.md
Section: [Reference](https://zshot-cli.com/reference/)

[`--crawl-links`](/reference/flags/archive/crawl-links) archives a site by following its links. A sitemap target archives the pages it lists, as [`--crawl-url-is-sitemap`](/reference/flags/archive/crawl-url-is-sitemap) controls. Both write WARC or WACZ.

## Pacing

Page loads on one host wait 1 second apart. [`--crawl-delay`](/reference/flags/archive/crawl-delay) sets the wait, and `0` removes it. A host whose robots.txt asks for a longer `Crawl-delay` is paced by that instead; [`--skip-robots-txt`](/reference/flags/browser/skip-robots-txt) ignores it.

A crawl runs one browser per CPU, and [`--crawl-concurrency`](/reference/flags/archive/crawl-concurrency) sets the count. The delay holds across every browser, so more browsers speed a crawl only across several hosts or with a shorter delay.

## Respect Site Ownership

ZShot follows robots.txt by default, its pacing included. Crawl at a rate the site's owner would accept.

## Related

Flags:

- [`--crawl-links`](https://zshot-cli.com/reference/flags/archive/crawl-links/)
- [`--crawl-url-is-sitemap`](https://zshot-cli.com/reference/flags/archive/crawl-url-is-sitemap/)
- [`--crawl-delay`](https://zshot-cli.com/reference/flags/archive/crawl-delay/)
- [`--crawl-concurrency`](https://zshot-cli.com/reference/flags/archive/crawl-concurrency/)
- [`--skip-robots-txt`](https://zshot-cli.com/reference/flags/browser/skip-robots-txt/)
- [`--crawl-link-depth`](https://zshot-cli.com/reference/flags/archive/crawl-link-depth/)
- [`--crawl-max-links`](https://zshot-cli.com/reference/flags/archive/crawl-max-links/)
- [`--crawl-allow-url`](https://zshot-cli.com/reference/flags/archive/crawl-allow-url/)
- [`--crawl-deny-url`](https://zshot-cli.com/reference/flags/archive/crawl-deny-url/)
- [`--crawl-link-selector`](https://zshot-cli.com/reference/flags/archive/crawl-link-selector/)
- [`--crawl-sitemap-max-depth`](https://zshot-cli.com/reference/flags/archive/crawl-sitemap-max-depth/)
- [`--crawl-media-sources`](https://zshot-cli.com/reference/flags/archive/crawl-media-sources/)
- [`--crawl-page-timeout`](https://zshot-cli.com/reference/flags/archive/crawl-page-timeout/)

Topics:

- [WACZ](https://zshot-cli.com/reference/formats/wacz/)
- [WARC](https://zshot-cli.com/reference/formats/warc/)
- [Crawl a site into a WACZ](https://zshot-cli.com/examples/crawl-links/)
- [Archive a sitemap into a WACZ](https://zshot-cli.com/examples/wacz-sitemap/)
- [robots.txt](https://en.wikipedia.org/wiki/Robots.txt) — Wikipedia

## Navigation

- [Examples](https://zshot-cli.com/examples/)
- [Download](https://zshot-cli.com/download/)
- [Purchase](https://zshot-cli.com/purchase/)
- [Reference](https://zshot-cli.com/reference/)
  - [Formats](https://zshot-cli.com/reference/formats/)
  - [Flags](https://zshot-cli.com/reference/flags/)
  - [Agents](https://zshot-cli.com/reference/agents/)
  - [Configuration](https://zshot-cli.com/reference/configuration/)
  - **Crawling**
  - [Docker](https://zshot-cli.com/reference/docker/)
  - [Exit Codes](https://zshot-cli.com/reference/exit_codes/)
  - [Linux](https://zshot-cli.com/reference/linux/)
  - [LLM Connections](https://zshot-cli.com/reference/llm_connections/)
  - [LLM Formatting](https://zshot-cli.com/reference/llm_formatting/)
  - [Local Inputs](https://zshot-cli.com/reference/inputs/)
  - [Navigator](https://zshot-cli.com/reference/navigator/)
  - [Security](https://zshot-cli.com/reference/security/)
  - [Server](https://zshot-cli.com/reference/server/)
  - [Shell completions](https://zshot-cli.com/reference/shell_completions/)
  - [Storage](https://zshot-cli.com/reference/storage/)
  - [Timeouts](https://zshot-cli.com/reference/timeouts/)
  - [Web Bot Auth](https://zshot-cli.com/reference/web_bot_auth/)
- [Support](https://zshot-cli.com/support/)
- [Changelog](https://zshot-cli.com/changelog/)

Site index: https://zshot-cli.com/llms.txt