On this page
01 Why signatures, and why now
Every bot identity check an operator has used so far is weak in one direction. A user-agent token is a claim anyone can copy. Forward-confirmed reverse DNS is strong but slow, and many new operators run from cloud ranges with generic reverse names. Published address lists go stale and cannot distinguish two services sharing a provider. As crawler and AI agent traffic grew, operators and site owners converged on a cryptographic answer: the client signs the request, and the site checks the signature against a key the operator publishes. This page in the network monitoring hub explains the mechanism.
02 How RFC 9421 signatures work
RFC 9421, published in 2024, defines a general way to sign parts of an HTTP message. The client chooses components to cover, for example the method, the authority (host), the path and a date, builds a canonical signature base string from them, and signs it with its private key. It sends two headers: Signature-Input, which lists the covered components, the key identifier, the algorithm and creation and expiry times, and Signature, which carries the signature bytes. Both use the Structured Field Values syntax, so parsers are strict and unambiguous.
GET /network-monitoring/sflow/ HTTP/1.1
Host: example.net
Signature-Input: sig1=("@authority" "@path");created=1790000000;expires=1790000300;keyid="k1";tag="web-bot-auth"
Signature: sig1=:base64-signature-value:03 Verifying a signed request
Find the key
Read the key identifier from Signature-Input and look it up in the operator's published key directory, cached on your side. The Web Bot Auth drafts describe how an operator advertises that directory, for example through a header that points at a well-known location.
Check the time window
Reject signatures whose creation time is in the future or whose expiry has passed. A short window, minutes rather than hours, limits replay.
Rebuild the signature base
From the actual request, extract exactly the components listed, in the listed order, and build the base string as RFC 9421 specifies. Any proxy that rewrites the host or path before this step will break verification, so verify at the first hop that sees the original request.
Verify and record
Check the signature with the public key and the stated algorithm. Record the operator, the key and the result in the access log so policy and reporting can use it.
Apply policy
A verified request gets the treatment your policy gives that operator; an unverified request claiming the same operator's user-agent is treated as a forgery.
04 What a valid signature proves
A valid signature proves that the request came from someone holding the operator's private key, and that the covered components were not changed on the way. It does not prove which end user asked an agent to act, whether the task is one you want, or that the operator respects robots.txt. It replaces the weakest link, identity, and leaves policy to you. Combine it with the robots.txt and llms.txt rules for AI clients for the honest crawlers and with rate limits for everything else.
05 Failure modes to plan for
- Key rotation. Cache keys with the lifetime the directory advertises and refetch on an unknown key identifier, or verification will fail every time an operator rotates.
- Intermediaries. TLS-terminating proxies that rewrite the authority or normalise paths break the base string; verify before rewriting.
- Clock skew. Allow a small tolerance on creation time, and keep your own clocks on NTP.
- Cost. Signature checks are cheap, but key fetches are not; never fetch a key directory inline for every request.
The older checks still have a place: forward-confirmed DNS and address lists, described in identifying AI crawlers at the edge, cover operators that do not sign yet.
06 Questions
What are HTTP message signatures?
A standard (RFC 9421) for signing selected parts of an HTTP request or response. The signer sends Signature-Input and Signature headers; the receiver rebuilds the signed string and verifies it with the signer's public key.
How do I verify that a bot is who it claims to be?
Where the operator signs requests, verify the signature against its published key. Otherwise use forward-confirmed reverse DNS or the operator's published address list. The user-agent string alone proves nothing.
What is Web Bot Auth?
An IETF working group standardising how bot and agent operators publish signing keys and how sites discover and use them with HTTP message signatures.
Does a valid signature mean the request is safe?
No. It proves which operator sent it and that it was not altered. Whether the action is allowed is still your policy decision.
Where should the check run?
At the first proxy or balancer that sees the original request, before anything rewrites the host or path.