On this page
01 What bot verification proves
Bot verification is the act of checking that a request which claims to come from a named crawler, Googlebot, Bingbot or an AI fetcher, really originates from that operator's infrastructure. The claim arrives in the User-Agent header, and anything can write that header. User-agent spoofing is the cheapest attack on the web: a scraper copies a well-known crawler's string and inherits whatever allowances your balancer grants it.
Verification replaces the claim with a fact drawn from something the impostor does not control: the DNS zone the operator runs, the address space it publishes, or a private signing key it holds. Each proves a different thing, and this page in the provenance lane index sets out how to use all three in one procedure, what to do with the result at the balancer, and how to keep a record that stands up later.
02 Reverse then forward DNS, the baseline check
Reverse DNS verification is the method the large search engines document first. The operator publishes PTR records for its crawler addresses under a domain it controls, and A or AAAA records that map those names back to the same addresses. The check has two halves and both are required. A PTR lookup alone proves nothing, because anyone who controls their own reverse zone can name a host crawl-1-2-3-4.googlebot.com.
The forward lookup closes the loop: resolve the returned name and require that the answer set contains the address you started with. Google's verification page states that the reverse name must end in googlebot.com, google.com or googleusercontent.com, and gives the two-command sequence below as the manual method. Bing documents an equivalent procedure on its verification page with its own hostname suffix; check the current page rather than a copied list, because suffixes change.
$ dig +short -x 66.249.66.1 crawl-66-249-66-1.googlebot.com. $ dig +short A crawl-66-249-66-1.googlebot.com 66.249.66.1 $ dig +short -x 2001:db8::1a2b ; no PTR: treat as unverified, fall through to the range list
The pair of lookups is slow, so it is never done inline on the request path; it runs once per new source address and the answer is cached with a bounded lifetime. A negative answer (no PTR, or a name outside the documented suffix) is a definite result: the client is unverified, whatever its user-agent says.
03 Published IP range feeds
The second source is the operator's published address list. Google publishes several JSON documents, one per crawler category (common crawlers, special-case crawlers, user-triggered fetchers), each listing prefixes in CIDR notation, and its documentation recommends them for large-scale lookups in place of DNS. Bing publishes a single JSON file of ipv4Prefix entries with a creation timestamp. Both are plain HTTPS fetches that a scheduled job can pull, parse and load into a prefix table on the balancer.
A range list answers a different question from DNS: not whether the operator names this address, but whether the address falls in space the operator has declared. The two usually agree, but the list lags when a crawler moves to new blocks, and a stale copy marks legitimate traffic unverified. Refresh at least daily, record the fetch time with the table, and alert when the feed fails to load, because a silent failure turns every verification into a miss.
04 Signed requests: RFC 9421 and Web Bot Auth
The third source is cryptographic. RFC 9421, HTTP Message Signatures, defines how a client signs selected components of a request (method, authority, path, chosen headers, a creation and expiry time) and carries the result in Signature-Input and Signature headers. The IETF Web Bot Auth working group is building a profile of it for automated clients: the current working group draft defines a Signature-Agent header that names the operator's key directory, a directory format based on JSON Web Key Sets, and a well-known path, /.well-known/http-message-signatures-directory, from which the directory is served.
A valid signature proves possession of the operator's private key, which neither DNS nor an address list can do, and it survives a change of source address. It is also the only method that works when the crawler runs from shared cloud space with generic reverse names. Google describes its own deployment as experimental and asks verifiers to fall back to DNS and address checks because not every request is signed; treat that as the general state of the art. The RFC 9421 verification walkthrough covers the signature base, key caching and the time window in detail.
05 Bot verification procedure at the balancer
The procedure assumes a layer 7 balancer or reverse proxy that sees the true client address. It yields one of three states per request: verified (identity confirmed by at least one method), unverified (a claim with no confirming evidence), or forged (a claim contradicted by evidence, for example a Googlebot user-agent from an address with a PTR in an unrelated zone).
Classify the claim
Match the
User-Agentagainst the short list of crawler tokens you care about. Requests that claim nothing skip verification and go to ordinary bot scoring; only claims are verified.Check for a signature first
If
Signature-InputandSignature-Agentare present, look the key up in your cached copy of the operator's directory, check the created and expires window, rebuild the signature base and verify. A pass is verified; a failure with a known operator is forged.Consult the address table
For unsigned claims, test the source address against the prefix table loaded from the operator's published feed. Record the feed's fetch time with the result.
Run forward-confirmed reverse DNS on a miss
If the address is not in the table, run the PTR lookup and the forward lookup asynchronously, cache the outcome for the record's TTL or a fixed hour, whichever is shorter, and apply it to subsequent requests from that address.
Apply policy by state
Verified clients get the operator's agreed rate and access. Unverified claims get the same treatment as anonymous automation. Forged claims get a hard limit or a block, since the forgery itself is evidence of intent.
Write the verification record
Log the claim, the method that decided it, the state, and the evidence (name returned, prefix matched, key identifier) on every request, not only failures.
06 Comparing the verification methods
| Method | What it proves | Cost per check | What it cannot prove |
|---|---|---|---|
| Forward-confirmed reverse DNS | The address is one the operator names in its own zones | Two lookups, cached per address | Nothing about the request content; fails for addresses with no PTR |
| Published IP range feed | The address lies in space the operator declares | One prefix table match | Freshness; a stale list misses new blocks and a shared block admits neighbours |
| RFC 9421 signature | The sender holds the operator's private key and the covered fields are intact | One signature verification, key cached | Which user asked, or that the operator honours robots.txt |
| User-agent string alone | Nothing; it is a self-declaration | None | Everything; treat as a claim to be checked |
07 A verification log as evidence
Log every outcome, not only blocks, because the log is what you will be asked for. An operator disputing a block or an internal review after a leak both come down to one question: at that time, from that address, what did the balancer decide and on what evidence? A line that carries the state and the evidence answers it; a line with only the user-agent does not.
Keep the record small and structured: timestamp with zone, client address, claimed token, method (sig, prefix, fcrdns, none), state, and one evidence field. Retention and integrity questions belong to the tamper-evident logging procedure. For traffic analysis rather than proof, the patterns in reading crawler fleets in access logs show what the verified and unverified populations tend to look like.
08 Pitfalls: cached PTRs, IPv6 and proxied origins
- Caching a PTR result past its life. Operators move crawlers between blocks. A cache that keeps a verified verdict for days will keep verifying an address after the operator has released it. Bound the cache by the DNS TTL and by a hard ceiling.
- IPv6 with no reverse zone. Many crawler ranges have PTR records only for IPv4. An IPv6 source with no PTR is not a forgery; it is a miss that must fall through to the prefix table, which for the large operators includes
ipv6Prefixentries. - Origins behind a CDN or proxy. An origin that verifies the TCP peer address is verifying the proxy, not the crawler. Verification has to run at the first hop that sees the client, or the origin must trust a forwarded-address header only from that hop's known addresses. This is the same trust problem that token buckets for unverified crawlers face when keyed by address.
- Blocking every unverified claim. Some legitimate fetchers, previews and archive crawlers have no PTR and no feed. Unverified means anonymous, and anonymous automation gets the ordinary policy, not a ban.
Verification is the first step of the lane collected under provenance on the wire: once a client is proven, mutual TLS on the edge-to-origin leg proves the other end of your own connection, and an audit log built to be tamper-evident keeps the result as a record.
09 Questions
How do I verify Googlebot?
Run a reverse DNS lookup on the source address, confirm the name ends in googlebot.com, google.com or googleusercontent.com, then resolve that name forward and require the original address in the answer. For bulk checks, match the address against the JSON prefix lists Google publishes for each crawler category.
How do I verify Bingbot?
The same two-step DNS check applies, using the hostname suffix Bing documents on its verification page, and Bing publishes a JSON file of its crawler prefixes with a creation timestamp. Load that file into the balancer's prefix table on a schedule and treat a stale copy as a warning.
Is a reverse DNS lookup enough on its own?
No. Anyone who controls the reverse zone for their own addresses can put any name in a PTR record. The forward lookup on that name, returning the original address, is what makes the check meaningful. Skip it and the check proves nothing.
What is Web Bot Auth?
An IETF working group profiling RFC 9421 HTTP Message Signatures for automated clients. Its draft protocol adds a Signature-Agent header naming the operator's key directory, a JWKS-based directory format and a well-known URI for it, so a site can verify a crawler by signature rather than by address.
Does bot verification tell me the crawler is safe?
It tells you who sent the request and nothing more. A verified crawler can still ignore robots.txt, fetch too fast or index what you would rather it did not. Verification settles identity; policy and rate limits remain your decisions.
Where should verification run in my stack?
At the first device that sees the real client address, normally the layer 7 balancer or edge proxy. An origin behind a CDN sees the proxy's address and cannot verify anything unless it trusts a forwarded-address header from that hop.
Bot verification is three cheap checks and a discipline about logging them. Set up the prefix feeds and the DNS cache first, add signature verification as operators publish keys, and then decide what verified and unverified clients are allowed to do.