SQLite

Output type: sqlite
Extension: .sqlite

LicensePro

A page or a crawl as one SQLite database you can search and query. It holds what a WARC holds — every exchange with its body, which page it belongs to, and which responses revisit a stored payload — beside each page’s rendered HTML, text and screenshot, and the links a crawl followed.

Table	Holds
`crawl`	The capture: the zshot version, target URL and command line
`pages`	Each page’s URL, title, HTML, `innerText`, screenshot and the page whose link led to it
`resources`	Each HTTP exchange, its headers as JSON
`payloads`	Each response body once, by digest
`page_resources`	Which page loaded each exchange, and whether it was served in full or as a revisit
`links`	Each link a crawled page offered, and what the crawl made of it
`pages_fts`, `payloads_fts`	Full-text search over page text and text bodies
`headers`	Each exchange’s headers, one per row, names lowercased

--sqlite PATH writes one beside any other output. latest.sql documents every column. /schemas/sqlite/<version>.sql holds each release’s.

Example

Run locally

zshot -f winnie-the-pooh.sqlite \
  --crawl-links \
  --crawl-link-depth 1 \
  https://zshot-cli.com/example_assets/winnie-the-pooh/

{
  "url": "https://zshot-cli.com/example_assets/winnie-the-pooh/",
  "output_type": "sqlite",
  "crawl_links": true,
  "crawl_link_depth": 1
}

curl -X POST http://127.0.0.1:3000/ \
  -H "Content-Type: application/json" \
  -d '{"url":"https://zshot-cli.com/example_assets/winnie-the-pooh/","output_type":"sqlite","crawl_links":true,"crawl_link_depth":1}' \
  -o winnie-the-pooh.sqlite

Queries

Pages that mention a word, with the matching text:

SELECT p.url, snippet(pages_fts, 1, '[', ']', '…', 10)
FROM pages_fts JOIN pages p ON p.id = pages_fts.rowid
WHERE pages_fts MATCH 'honey';

What one page loaded, and how:

SELECT resource_url, status, served, revisit_reason, size
FROM page_resources_v WHERE page_url LIKE '%/chapter-1%';

Responses a server told caches not to store:

SELECT r.url, h.value
FROM headers h JOIN resources r ON r.id = h.resource_id
WHERE h.direction = 'response' AND h.name = 'cache-control' AND h.value LIKE '%no-store%';

Each revisit, and the page that stored its payload:

SELECT page_url, resource_url, revisit_reason, original_page_id FROM revisits_v;