Web Bot Auth
Web Bot Auth lets a site verify which automated agent sent a request. The agent signs each request with a private key and publishes the matching public keys at its own origin; a verifier such as Cloudflare checks the signature against them and can then treat the agent’s traffic as known rather than anonymous. --web-bot-auth makes ZShot sign as an agent.
Setup
- Generate an Ed25519 key:
openssl genpkey -algorithm ed25519 -out bot-key.pem. - Publish its public key in a key directory at
https://<agent origin>/.well-known/http-message-signatures-directory. - Register the agent with each verifier that should trust it.
- Render with
--web-bot-auth https://<agent origin>=bot-key.pem.
The agent origin is the identity a site sees, so it is an https:// origin you control, with no path, query or credentials.
Registration
A verifier trusts the key once the agent is registered with it. For Cloudflare, submit the agent’s key directory URL in the Bot Submission Form with Request Signature as the verification method. Cloudflare reads the directory only when its response is signed by each key it lists.
Signed Requests
Each request a render sends carries three headers, signed for the host it goes to. That includes subresources and ZShot’s own robots.txt and sitemap fetches.
| Header | Holds |
|---|---|
Signature-Agent | The agent origin, as a quoted string |
Signature-Input | The covered components, @authority and signature-agent, with created, expires, keyid, alg="ed25519", a random nonce and tag="web-bot-auth" |
Signature | The Ed25519 signature, labelled sig1 |
The format follows RFC 9421 HTTP Message Signatures, as Cloudflare verifies them. keyid is the key’s RFC 7638 JWK thumbprint. A signature expires 60 seconds after it is made, so each request is signed when it is sent.
Keys on a Server
A server started with --web-bot-auth signs every request it renders with its key. A request’s web_bot_auth field replaces that key for that request, so one server can sign as several agents. A request can give the key only as a data: URI or an http(s) URL; the server reads no key file a request names.
A key URL is fetched when each render starts, so a rotated key takes effect on the next render without restarting the server.