robots.txt and llms.txt: Telling AI Agents What to Fetch

01 /Two files, two jobs

Operators who ran web farms behind a ServerIron in 2002 knew robots.txt as the file that kept search spiders out of cgi-bin. It still does that job, and it is now also the first thing an AI training crawler or answer-engine fetcher reads. Alongside it sits a newer convention, /llms.txt, which does something different: instead of saying where a client may not go, it says where a language-model tool should start. This page, part of the network monitoring section, explains what each file can express and where each stops.

02 /How a robots.txt group is matched

RFC 9309 turned the old convention into a standard in 2022. A file is a series of groups. Each group starts with one or more User-agent lines and continues with Allow and Disallow rules. A crawler picks the group whose user-agent value matches its product token, case-insensitively; if none matches it uses the * group. Several user-agent lines can share one group, which keeps a long allowlist short. Within the chosen group the longest matching path decides, and when an Allow and a Disallow match with equal length, Allow wins.

The standard also sets practical limits. Crawlers must read at least 500 kibibytes of the file, may cache it for up to a day, and should treat a 4xx response as "no restrictions" and a 5xx as "assume everything is disallowed" for a while. That last rule matters when a balancer health check fails: a robots.txt served from a broken pool can silently stop every compliant crawler. Monitor it like any other endpoint.

03 /Control tokens and usage wishes

Several operators publish tokens that appear only in robots.txt and never in a request, used to say whether content already fetched by a general crawler may be used for model training. They work only because the operator chose to honour them. The IETF AI Preferences working group is drafting a common vocabulary for this kind of statement so that one line can express a preference to every operator at once. It is still a statement of preference: the file is read by the crawler, not checked by the network. The guide to identifying AI crawler traffic lists the tokens seen in requests and how to verify them.

04 /What llms.txt adds

The llms.txt proposal, published in 2024, is a Markdown file at /llms.txt: a title, a one-paragraph summary, and sections of links with short descriptions. It is written for tools that fetch a few pages at question time and need to know which ones matter. It carries no access rules at all. A good llms.txt lists the canonical pages, groups them the way the site is organised, and stays current; a stale one sends tools to pages that redirect. Think of it as a hand-written sitemap for readers that summarise rather than index.

$ curl -s https://example.net/robots.txt
User-agent: *
Allow: /
Disallow: /staging/
Sitemap: https://example.net/sitemap.xml
$ curl -s https://example.net/llms.txt | head -2
# Example Net
> Reference guides for campus network operators.

05 /Serving both files from the edge

Both files are tiny and requested constantly, so serve them from cache with a short lifetime and a correct text/plain type. Keep them out of any bot-management rule that might return a challenge page instead of the file; a crawler that receives HTML where it expected robots.txt may treat the site as unrestricted or as blocked, depending on its parser. Log requests to both paths separately. A client that never fetches robots.txt but claims a crawler token is worth a second look, and the cryptographic bot verification methods now emerging give operators a stronger check than the file ever could.

For the enforcement half of the problem, the fingerprinting signals for automated clients and the balancer-side controls in an organisation-wide crawler policy pick up where these files stop.