AI Agent Traffic: When a Browser Drives Itself

01 /Crawler, fetcher, agent

The network monitoring section already covers crawlers, which walk a site to copy it, and answer-engine fetchers, which pull a few pages to summarise them. A third client class has arrived: the agent. An agent is a language model given a browser and a goal ("book the 9:40 train", "download last month's invoice") that clicks and types its way through a site to reach it. Its traffic is closer to a human session than to a crawl, and most of the old heuristics fail on it.

02 /What agent sessions look like

On the wire an agent session is a normal TLS connection from a real browser engine, so the ClientHello and HTTP/2 settings match a mainstream browser. The differences show up in behaviour and origin. Paths are linear: the agent goes straight from landing page to search to result to form, without the back-and-forth of a person comparing options. Time between actions is regular and short, then pauses for several seconds while the model decides. Sessions often start from cloud provider address space rather than residential or mobile networks. Pointer movement, where a site measures it, is absent or synthetic. The fingerprinting methods for automated clients describe the individual signals; for agents, the session is the unit to score.

03 /Measuring it

Count agent sessions, not agent requests. Build sessions from balancer or application logs keyed on the session cookie, then flag those with a datacentre origin, a linear path and uniform think time. Compare the conversion events (logins, posts, purchases) in flagged sessions with the rest. The numbers matter because agent traffic concentrates on exactly the endpoints that change state, where a scripted mistake or abuse costs most. Packet sampling with sFlow will not see any of this; it lives above TLS.

04 /Declared agents and signed requests

Some agent operators now identify their traffic. The first generation used a user-agent string, which anyone can copy. The emerging approach signs each request with a key the operator publishes, using HTTP Message Signatures (RFC 9421); the IETF Web Bot Auth working group is standardising how a site discovers and checks those keys. A signed request proves which operator sent it, though not which user asked for the task. The guide to verifying bots with HTTP message signatures walks through the check.

05 /Setting policy per action

Because agents act for real users, blocking them wholesale can break legitimate use: a person who asked an assistant to download their own invoice is still a customer. Set rules per action instead. Reading public pages is usually fine. Logging in may be allowed for declared, verified agents. Purchases, password changes and bulk exports deserve a step a model cannot complete alone, such as a one-time code sent to the account holder. Rate-limit form posts per session and per origin network, the same token-bucket approach as rate limiting crawlers at the balancer, but applied to state-changing endpoints.

Referral data completes the picture. Visits that arrive after an assistant cited your page are a different stream again, covered in measuring referral traffic from AI answers.