AI Crawler Traffic: Identifying LLM Bots at the Network Edge

01 /What counts as AI crawler traffic

Two kinds of automated client now sit alongside the classic search indexers. Training crawlers fetch pages in bulk to build text corpora. Answer-engine fetchers retrieve a handful of pages on demand when a user asks a question, then summarise them. Both arrive as ordinary HTTP over TLS. To a router they are just flows; to a reverse proxy they are requests with a user-agent string. This page belongs to the network monitoring hub and treats the problem the way an operator would: identify, verify, measure, then set policy.

02 /User-agent tokens and why they are claims

Most operators publish a product token that appears in the User-Agent header, and the same token is what robots.txt rules match against. Some tokens exist only in robots.txt: they control how content already fetched by a general crawler may be used, and never appear in a request.

Any client can send any header. Scrapers routinely borrow a well-known crawler token because some sites wave it through. A log line reading CCBot/2.0 proves only that someone typed CCBot. Treat the token as a label to verify, never as an identity.

03 /Verifying a crawler: reverse DNS and published ranges

There are two reliable checks, and serious crawler operators support at least one. The first is forward-confirmed reverse DNS (FCrDNS). Look up the PTR record of the source address, check that the name falls under the operator's documented domain, then resolve that name forward and confirm it returns the original address. The forward step matters: whoever controls the reverse zone for an address can make its PTR say anything.

$ dig -x 203.0.113.45 +short
crawl-203-0-113-45.crawler.example.com.
$ dig crawl-203-0-113-45.crawler.example.com A +short
203.0.113.45
# suffix matches the documented domain and the address round-trips: verified

The second check is a published address list. Several operators publish their crawler prefixes as a machine-readable file. Fetch it on a schedule, load the prefixes into an address set on the proxy or balancer, and match source addresses against it. Cache results from either method; running two DNS lookups per request adds latency and loads your resolvers.

04 /robots.txt semantics and its limits

RFC 9309 standardised the Robots Exclusion Protocol in 2022. A crawler reads /robots.txt, finds the group whose user-agent line matches its token (or *), and applies the most specific Allow or Disallow path rule. That is the entire protocol. It has no authentication, no rate field in the standard and no penalty. A compliant crawler obeys it; a non-compliant one ignores it and you learn nothing. Newer vocabularies for expressing AI usage preferences are being drafted at the IETF, but they share the same property: they express a wish.

05 /What the traffic looks like in logs and sFlow

In access logs a crawler has a recognisable shape. It fetches /robots.txt and sitemaps early and repeatedly. It requests HTML but not the CSS, scripts, images and fonts a browser pulls with every page. It rarely sends cookies or a referrer. Request timing is regular, often in bursts from many addresses in one prefix, and it walks deep archive URLs that humans never visit. The signals that separate automated clients from people go further than this, down to TLS and TCP.

Packet sampling sees less. sFlow samples carry only the first bytes of each sampled frame, and with TLS the HTTP headers are encrypted, so no user-agent is visible. What sFlow does well is volume: bytes and flows by source prefix and origin AS, fast enough to spot a new crawler ramping up. Use it to find the heavy prefixes, then read proxy logs for identity. A SPAN session into a capture host fills the gap only where you terminate TLS yourself.

06 /Measuring crawler share of traffic

Compute share on three axes, because they disagree: requests, bytes served and origin CPU time. Crawlers skip assets, so their share of bytes is usually lower than their share of requests, while their share of origin work can be higher because they hit uncached deep pages. Split each figure into verified crawler, unverified crawler label and everything else. The unverified bucket is the one that grows when scrapers learn which tokens you allow.

07 /Allow, shape or block

Decide per verified crawler, not per category. Allow crawlers whose use of the content you accept. Shape those you accept but whose rate hurts the origin, with a per-identity limit on the balancer; the crawler rate limiting procedure covers token buckets and 429 responses. Block the rest, and block unverified clients that claim a crawler token outright, since a forged identity has no legitimate reason to exist. Enforcement at Layer 7 needs a device that reads headers, a trade described under what each balancer layer can see.

The files crawlers read first are covered in how robots.txt and llms.txt speak to AI clients; the stronger identity check is signature-based bot verification, and the reader visits that follow a citation are measured in answer-engine referral measurement.