AI Agent Traffic: When a Browser Drives Itself

01 /Crawler, fetcher, agent

The network monitoring section already covers crawlers, which walk a site to copy it, and answer-engine fetchers, which pull a few pages to summarise them. A third client class has arrived: the agent. An agent is a language model given a browser and a goal ("book the 9:40 train", "download last month's invoice") that clicks and types its way through a site to reach it. Its traffic is closer to a human session than to a crawl, and most of the old heuristics fail on it.

02 /What agent sessions look like

On the wire an agent session is a normal TLS connection from a real browser engine, so the ClientHello and HTTP/2 settings match a mainstream browser. The differences show up in behaviour and origin. Paths are linear: the agent goes straight from landing page to search to result to form, without the back-and-forth of a person comparing options. Time between actions is regular and short, then pauses for several seconds while the model decides. Sessions often start from cloud provider address space rather than residential or mobile networks. Pointer movement, where a site measures it, is absent or synthetic. The fingerprinting methods for automated clients describe the individual signals; for agents, the session is the unit to score.

03 /Measuring it

Count agent sessions, not agent requests. Build sessions from balancer or application logs keyed on the session cookie, then flag those with a datacentre origin, a linear path and uniform think time. Compare the conversion events (logins, posts, purchases) in flagged sessions with the rest. The numbers matter because agent traffic concentrates on exactly the endpoints that change state, where a scripted mistake or abuse costs most. Packet sampling with sFlow will not see any of this; it lives above TLS.

04 /A worked example of AI agent traffic in the logs

The session below is reconstructed from a balancer access log keyed on the session cookie. Static assets are served from a separate hostname and do not appear; what remains is the HTML and form traffic, which is where the shape of AI agent traffic shows. The addresses come from the documentation range reserved by RFC 5737 and the timings are illustrative.

$ awk '$3=="sid=7f3a"' access.log
09:12:03 203.0.113.40 sid=7f3a GET  /                              200 0.02s ref=-
09:12:05 203.0.113.40 sid=7f3a GET  /search?q=invoice+august        200 0.05s ref=/
09:12:07 203.0.113.40 sid=7f3a GET  /account/login                 200 0.02s ref=/search
09:12:14 203.0.113.40 sid=7f3a POST /account/login                 302 0.11s ref=/account/login
09:12:16 203.0.113.40 sid=7f3a GET  /account/invoices              200 0.04s ref=/account/login
09:12:18 203.0.113.40 sid=7f3a GET  /account/invoices/2026-08.pdf  200 0.09s ref=/account/invoices
$ whois 203.0.113.40 | grep -i 'netname\|descr'
netname: EXAMPLE-CLOUD-COMPUTE-EU-WEST
descr:   Compute instances, dynamic assignment

Six requests in fifteen seconds, every gap close to two seconds except one seven-second pause before the login POST, which is the model reading the form and deciding what to type. The path has no detour: landing page, one search, one login, one document. The origin is a compute range with a generic description rather than a residential or mobile allocation. No single line is conclusive. Together they score well above any human baseline for this site, and the session ends with exactly the kind of action (a document download behind a login) that the policy section treats as a decision point.

Two things make the example usable in practice. The session key must be present on the first request, which means the cookie has to be set by the balancer or edge, not by the application after the first page; otherwise the landing request cannot be joined to the rest. And the origin lookup must be cached and done offline against a table of allocations, never inline per request. Where an operator signs its requests, the verified identity replaces most of this scoring; the bot verification procedure in the provenance lane orders the checks from weakest to strongest.

05 /Declared agents and signed requests

Some agent operators now identify their traffic. The first generation used a user-agent string, which anyone can copy. The emerging approach signs each request with a key the operator publishes, using HTTP Message Signatures (RFC 9421); the IETF Web Bot Auth working group is standardising how a site discovers and checks those keys. A signed request proves which operator sent it, though not which user asked for the task. The guide to verifying bots with HTTP message signatures walks through the check.

06 /Setting policy per action

Because agents act for real users, blocking them wholesale can break legitimate use: a person who asked an assistant to download their own invoice is still a customer. Set rules per action instead. Reading public pages is usually fine. Logging in may be allowed for declared, verified agents. Purchases, password changes and bulk exports deserve a step a model cannot complete alone, such as a one-time code sent to the account holder. Rate-limit form posts per session and per origin network, the same token-bucket approach as rate limiting crawlers at the balancer, but applied to state-changing endpoints.

Referral data completes the picture. Visits that arrive after an assistant cited your page are a different stream again, covered in measuring referral traffic from AI answers.

07 /Pitfalls when measuring agent sessions

• Counting requests instead of sessions. One agent stuck in a retry loop can post a form two hundred times in a minute. In a request count that looks like a small crawler; in a session count it is one broken client, which is the truth.
• Treating a cloud origin as proof. Remote desktops, corporate VPN exits and privacy relays also leave from compute ranges. Origin raises the score; it does not settle it.
• Filtering on the user-agent string. An agent driving a real browser sends the browser's own string. Declared agents add signature headers on top; undeclared ones are indistinguishable from the browser by that field alone.
• Keying rate limits on the source address. An operator's whole fleet may leave through a few NAT addresses, so one address limit throttles every user of that operator at once. Key limits on session and, where verified, on operator identity.
• Sampling the log. A one-in-N sample at the balancer breaks sessions apart. Log form posts and login endpoints at full rate even if page views are sampled.
• Dropping the logs after a month. A dispute about who submitted a form arrives long after the fact. Retention and integrity rules for access logs that can stand as evidence apply to agent sessions exactly as they do to any other request.

08 /Operator tips for agent traffic

• Add a verified_operator field to the access log, filled by the signature check at the first hop and left empty otherwise. Every downstream report can then split declared agents from the rest without re-parsing headers.
• For one week, capture full request bodies from flagged sessions on the two or three endpoints that change state. Reading fifty real agent form posts teaches more about failure shapes than any heuristic.
• When you refuse an agent action, return a status and a short plain-text reason the model can read (for example 403 with a one-line explanation). Agents retry silent failures; they tend to stop, or ask the user, on an explicit one.
• Write the rules down. A written policy that names agents as a class gives the balancer team, the application team and the people who answer complaints one document to point at.
• Review the origin allocation table and the operator key directories on a schedule. Both change without notice, and a stale table quietly turns declared agents back into anonymous ones.