AI Crawler Traffic: Identifying LLM Bots at the Network Edge

01 /What counts as AI crawler traffic

Two kinds of automated client now sit alongside the classic search indexers. Training crawlers fetch pages in bulk to build text corpora. Answer-engine fetchers retrieve a handful of pages on demand when a user asks a question, then summarise them. Both arrive as ordinary HTTP over TLS. To a router they are just flows; to a reverse proxy they are requests with a user-agent string. This page belongs to the network monitoring hub and treats the problem the way an operator would: identify, verify, measure, then set policy.

02 /User-agent tokens and why they are claims

Most operators publish a product token that appears in the User-Agent header, and the same token is what robots.txt rules match against. Some tokens exist only in robots.txt: they control how content already fetched by a general crawler may be used, and never appear in a request.

Any client can send any header. Scrapers routinely borrow a well-known crawler token because some sites wave it through. A log line reading CCBot/2.0 proves only that someone typed CCBot. Treat the token as a label to verify, never as an identity.

03 /Verifying a crawler: reverse DNS and published ranges

There are two reliable checks, and serious crawler operators support at least one. The first is forward-confirmed reverse DNS (FCrDNS). Look up the PTR record of the source address, check that the name falls under the operator's documented domain, then resolve that name forward and confirm it returns the original address. The forward step matters: whoever controls the reverse zone for an address can make its PTR say anything.

$ dig -x 203.0.113.45 +short
crawl-203-0-113-45.crawler.example.com.
$ dig crawl-203-0-113-45.crawler.example.com A +short
203.0.113.45
# suffix matches the documented domain and the address round-trips: verified

The second check is a published address list. Several operators publish their crawler prefixes as a machine-readable file. Fetch it on a schedule, load the prefixes into an address set on the proxy or balancer, and match source addresses against it. Cache results from either method; running two DNS lookups per request adds latency and loads your resolvers.

04 /robots.txt semantics and its limits

RFC 9309 standardised the Robots Exclusion Protocol in 2022. A crawler reads /robots.txt, finds the group whose user-agent line matches its token (or *), and applies the most specific Allow or Disallow path rule. That is the entire protocol. It has no authentication, no rate field in the standard and no penalty. A compliant crawler obeys it; a non-compliant one ignores it and you learn nothing. Newer vocabularies for expressing AI usage preferences are being drafted at the IETF, but they share the same property: they express a wish.

05 /What the traffic looks like in logs and sFlow

In access logs a crawler has a recognisable shape. It fetches /robots.txt and sitemaps early and repeatedly. It requests HTML but not the CSS, scripts, images and fonts a browser pulls with every page. It rarely sends cookies or a referrer. Request timing is regular, often in bursts from many addresses in one prefix, and it walks deep archive URLs that humans never visit. The signals that separate automated clients from people go further than this, down to TLS and TCP.

Packet sampling sees less. sFlow samples carry only the first bytes of each sampled frame, and with TLS the HTTP headers are encrypted, so no user-agent is visible. What sFlow does well is volume: bytes and flows by source prefix and origin AS, fast enough to spot a new crawler ramping up. Use it to find the heavy prefixes, then read proxy logs for identity. A SPAN session into a capture host fills the gap only where you terminate TLS yourself.

06 /What a log line proves about AI crawlers

An access log line is a statement by your own server: at this time, a client from this address asked for this resource with these headers, and the server answered with this status and this many bytes. That much it proves, provided the clock was right and the file has not been edited. It does not prove who the client was. The user-agent is the client's claim, and the address is only an address until the verification step has been run and its result written down next to the line.

203.0.113.45 [24/Sep/2026:09:14:02 +0000] "GET /robots.txt HTTP/1.1" 200 412 "-" "ExampleBot/1.0 (+https://crawler.example.com/bot)" rdns=crawl-203-0-113-45.crawler.example.com fcrdns=ok list=hit
203.0.113.45 [24/Sep/2026:09:14:03 +0000] "GET /switching/vlans/ HTTP/1.1" 200 18234 "-" "ExampleBot/1.0 (+https://crawler.example.com/bot)" rdns=crawl-203-0-113-45.crawler.example.com fcrdns=ok list=hit
198.51.100.77 [24/Sep/2026:09:14:03 +0000] "GET /switching/vlans/ HTTP/1.1" 200 18234 "-" "ExampleBot/1.0 (+https://crawler.example.com/bot)" rdns=- fcrdns=fail list=miss

The first two lines record a verified crawler: the PTR name falls under the operator's domain, the forward lookup returned the same address, and the address was in the published list at the time. The third line records an impostor using the same token. Without the appended fields, the three lines are identical except for the address, and a reader a month later has no way to tell them apart, because PTR records and published lists change. Verification is evidence only when it was performed at the time and recorded with the request.

Two things turn a log from a debugging aid into a record that stands up. The first is verification recorded inline, as above; the mechanics of each check, from reverse DNS to signed requests, are set out under how a bot proves its identity to the edge. The second is integrity: synchronised clocks, append-only storage, a hash chain or signature over each rotated file, and a retention period chosen in advance, which is the subject of keeping edge logs as tamper-evident evidence. Both belong to the authenticity at the network layer hub, where this page hands over.

07 /Measuring crawler share of traffic

Compute share on three axes, because they disagree: requests, bytes served and origin CPU time. Crawlers skip assets, so their share of bytes is usually lower than their share of requests, while their share of origin work can be higher because they hit uncached deep pages. Split each figure into verified crawler, unverified crawler label and everything else. The unverified bucket is the one that grows when scrapers learn which tokens you allow.

08 /Allow, shape or block

Decide per verified crawler, not per category. Allow crawlers whose use of the content you accept. Shape those you accept but whose rate hurts the origin, with a per-identity limit on the balancer; the crawler rate limiting procedure covers token buckets and 429 responses. Block the rest, and block unverified clients that claim a crawler token outright, since a forged identity has no legitimate reason to exist. Enforcement at Layer 7 needs a device that reads headers, a trade described under what each balancer layer can see.

09 /Expert tips

• Verify at the edge, not in a batch. A nightly job that re-resolves yesterday's addresses records today's DNS. Run the checks at request time and cache the result for hours, not days.
• Log the token and the verdict separately. A field that says "verified crawler" without the raw user-agent and the PTR name cannot be re-examined when the reference data changes.
• Watch the unverified bucket by token. A rise in failed verification for one crawler name means either that operator added ranges you have not fetched, or that scrapers have adopted the name. The fetch schedule for published lists tells you which.
• Keep a control. Fetch your own site from a cloud host with a borrowed crawler token once a day, and confirm the log marks it as unverified. A verification pipeline that never fails has probably stopped running.

The files crawlers read first are covered in how robots.txt and llms.txt speak to AI clients; the stronger identity check is signature-based bot verification, and the reader visits that follow a citation are measured in answer-engine referral measurement.