Verifying Bots With HTTP Message Signatures

01 /Why signatures, and why now

Every bot identity check an operator has used so far is weak in one direction. A user-agent token is a claim anyone can copy. Forward-confirmed reverse DNS is strong but slow, and many new operators run from cloud ranges with generic reverse names. Published address lists go stale and cannot distinguish two services sharing a provider. As crawler and AI agent traffic grew, operators and site owners converged on a cryptographic answer: the client signs the request, and the site checks the signature against a key the operator publishes. This page in the network monitoring hub explains the mechanism.

02 /How RFC 9421 signatures work

RFC 9421, published in 2024, defines a general way to sign parts of an HTTP message. The client chooses components to cover, for example the method, the authority (host), the path and a date, builds a canonical signature base string from them, and signs it with its private key. It sends two headers: Signature-Input, which lists the covered components, the key identifier, the algorithm and creation and expiry times, and Signature, which carries the signature bytes. Both use the Structured Field Values syntax, so parsers are strict and unambiguous.

GET /network-monitoring/sflow/ HTTP/1.1
Host: example.net
Signature-Input: sig1=("@authority" "@path");created=1790000000;expires=1790000300;keyid="k1";tag="web-bot-auth"
Signature: sig1=:base64-signature-value:

03 /Verifying a signed request

1. 01

   Find the key

   Read the key identifier from Signature-Input and look it up in the operator's published key directory, cached on your side. The Web Bot Auth drafts describe how an operator advertises that directory, for example through a header that points at a well-known location.

2. 02

   Check the time window

   Reject signatures whose creation time is in the future or whose expiry has passed. A short window, minutes rather than hours, limits replay.

3. 03

   Rebuild the signature base

   From the actual request, extract exactly the components listed, in the listed order, and build the base string as RFC 9421 specifies. Any proxy that rewrites the host or path before this step will break verification, so verify at the first hop that sees the original request.

4. 04

   Verify and record

   Check the signature with the public key and the stated algorithm. Record the operator, the key and the result in the access log so policy and reporting can use it.

5. 05

   Apply policy

   A verified request gets the treatment your policy gives that operator; an unverified request claiming the same operator's user-agent is treated as a forgery.

04 /Web Bot Auth: a signed-request check, step by step

The Web Bot Auth architecture draft profiles RFC 9421 for automated clients so that every operator signs the same way. The covered components must include @authority. The signature parameters must carry created, expires, a keyid that is the JWK thumbprint of the signing key, and tag="web-bot-auth"; a nonce is recommended against replay and the draft advises an expiry of no more than 24 hours. A Signature-Agent header names the operator's origin, and if present it must itself be a covered component. The companion directory draft serves the operator's public keys as a JSON Web Key Set at /.well-known/http-message-signatures-directory, with a cache lifetime in its own headers. An origin that receives an unsigned request from a client it wants to verify can ask for one with the Accept-Signature field from RFC 9421 section 5.

$ curl -s https://signer.example/.well-known/http-message-signatures-directory | jq -c '.keys[] | {kty, crv, kid}'
{"kty":"OKP","crv":"Ed25519","kid":"poqkLGiymh"}
$ grep -i '^signature' request.txt
Signature-Agent: "https://signer.example"
Signature-Input: sig1=("@authority" "signature-agent");created=1790000000;expires=1790003600;keyid="poqkLGiymh";tag="web-bot-auth";nonce="e8QxT2m9"
Signature: sig1=:MEUCIQDillustrativeSignatureBytes:
$ sigbase --from request.txt --sig sig1 > base.txt   # rebuilds the RFC 9421 signature base
$ cat base.txt
"@authority": example.net
"signature-agent": "https://signer.example"
"@signature-params": ("@authority" "signature-agent");created=1790000000;expires=1790003600;keyid="poqkLGiymh";tag="web-bot-auth";nonce="e8QxT2m9"
$ sigverify --jwk key.jwk --alg ed25519 --base base.txt --sig sig1.bin
OK  valid; created 12 s ago; expires in 3588 s; keyid matches thumbprint; nonce not seen before

The tools named are stand-ins for whatever your proxy or verification library provides; the sequence is what matters. Fetch the directory once and cache it for the advertised lifetime. Confirm the key identifier is the thumbprint of a key in that set. Rebuild the base from the request you actually received, before any rewriting. Verify, then check the window and the nonce. A failure at any step is logged with its reason, and a request that fails is handled as an anonymous client, not as the operator it named.

05 /When a verified HTTP message signature becomes evidence

A verified signature is a fact about one request at one moment. It becomes evidence only if the record you keep lets someone else reach the same conclusion later, after the key has rotated and the directory has changed. That means logging more than a pass or fail flag. The table lists the minimum. Store the fetched key itself, or its full JWK, because a thumbprint alone cannot be re-verified once the operator removes the key from its directory.

Two other pages in the provenance on the wire hub complete the chain. Signature checks sit at the top of a ladder that also includes reverse DNS and published address ranges for operators that do not sign yet; the wider bot verification method orders those checks and says which to trust. And the log line itself needs protecting: hashing each batch, signing the hash and keeping it under a retention rule is the subject of treating edge logs as evidence. A verified signature in an unsigned, rewritable log proves nothing to anyone but you.

06 /Signing responses: what the origin serves

RFC 9421 signs HTTP messages, not only requests. A response can cover the @status derived component, its own header fields and a Content-Digest of the body (RFC 9530), and it can bind itself to the request that caused it by naming request components with the ;req parameter, as in section 2.4 of the RFC. The origin signs with a key it publishes, in the same directory format a bot operator uses. The result is a page, image or document that carries proof of which origin produced it and that the bytes match, whatever path they took to arrive.

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Digest: sha-256=:illustrativeBodyDigest=:
Signature-Input: resp=("@status" "content-digest" "content-type" "@authority";req "@path";req);created=1790000000;keyid="origin-2026-09"
Signature: resp=:illustrativeSignatureBytes:

Three uses follow. Between an edge and its origin, a signed response lets the edge prove that what it cached is what the origin sent, which complements transport protection rather than replacing it. For a site that expects to be quoted, a signed response plus a retained copy is evidence that a page had a given digest at a given time, independent of any archive. And for a crawler or agent operator, a signed response is the only way to know that an intermediary did not alter what it fetched.

• Sign the final bytes. Any later compression, minification or image transcoding changes the body and breaks the digest. Sign after the last transformation, at the component that serves the object.
• Caches must keep the headers. A cache that strips or rewrites Signature-Input serves an object that can no longer be checked. Treat the two signature fields and Content-Digest as part of the object.
• Choose the covered set deliberately. Covering Date gives freshness; covering @path;req stops one signed body being replayed for another URL. Uncovered fields are unprotected by design.
• Publish and rotate keys like an operator. Serve your origin keys at the well-known directory path with a sensible cache lifetime, and overlap old and new keys through the longest cache period downstream.

07 /What a valid signature proves

A valid signature proves that the request came from someone holding the operator's private key, and that the covered components were not changed on the way. It does not prove which end user asked an agent to act, whether the task is one you want, or that the operator respects robots.txt. It replaces the weakest link, identity, and leaves policy to you. Combine it with the robots.txt and llms.txt rules for AI clients for the honest crawlers and with rate limits for everything else.

08 /Failure modes to plan for

• Key rotation. Cache keys with the lifetime the directory advertises and refetch on an unknown key identifier, or verification will fail every time an operator rotates.
• Intermediaries. TLS-terminating proxies that rewrite the authority or normalise paths break the base string; verify before rewriting.
• Clock skew. Allow a small tolerance on creation time, and keep your own clocks on NTP.
• Cost. Signature checks are cheap, but key fetches are not; never fetch a key directory inline for every request.

The older checks still have a place: forward-confirmed DNS and address lists, described in identifying AI crawlers at the edge, cover operators that do not sign yet.