robots.txt and llms.txt: Telling AI Agents What to Fetch

01 /Two files, two jobs

Operators who ran web farms behind a ServerIron in 2002 knew robots.txt as the file that kept search spiders out of cgi-bin. It still does that job, and it is now also the first thing an AI training crawler or answer-engine fetcher reads. Alongside it sits a newer convention, /llms.txt, which does something different: instead of saying where a client may not go, it says where a language-model tool should start. This page, part of the network monitoring section, explains what each file can express and where each stops.

02 /How a robots.txt group is matched

RFC 9309 turned the old convention into a standard in 2022. A file is a series of groups. Each group starts with one or more User-agent lines and continues with Allow and Disallow rules. A crawler picks the group whose user-agent value matches its product token, case-insensitively; if none matches it uses the * group. Several user-agent lines can share one group, which keeps a long allowlist short. Within the chosen group the longest matching path decides, and when an Allow and a Disallow match with equal length, Allow wins.

The standard also sets practical limits. Crawlers must read at least 500 kibibytes of the file, may cache it for up to a day, and should treat a 4xx response as "no restrictions" and a 5xx as "assume everything is disallowed" for a while. That last rule matters when a balancer health check fails: a robots.txt served from a broken pool can silently stop every compliant crawler. Monitor it like any other endpoint.

03 /Control tokens and usage wishes

Several operators publish tokens that appear only in robots.txt and never in a request, used to say whether content already fetched by a general crawler may be used for model training. They work only because the operator chose to honour them. The IETF AI Preferences working group is drafting a common vocabulary for this kind of statement so that one line can express a preference to every operator at once. It is still a statement of preference: the file is read by the crawler, not checked by the network. The guide to identifying AI crawler traffic lists the tokens seen in requests and how to verify them.

04 /What llms.txt adds

The llms.txt proposal, published in 2024, is a Markdown file at /llms.txt: a title, a one-paragraph summary, and sections of links with short descriptions. It is written for tools that fetch a few pages at question time and need to know which ones matter. It carries no access rules at all. A good llms.txt lists the canonical pages, groups them the way the site is organised, and stays current; a stale one sends tools to pages that redirect. Think of it as a hand-written sitemap for readers that summarise rather than index.

$ curl -s https://example.net/robots.txt
User-agent: *
Allow: /
Disallow: /staging/
Sitemap: https://example.net/sitemap.xml
$ curl -s https://example.net/llms.txt | head -2
# Example Net
> Reference guides for campus network operators.

05 /A worked llms.txt for a reference site

The proposal's format is strict enough to parse and loose enough to write by hand. In order: an H1 with the site or project name, which is the only required part; a blockquote with a one-paragraph summary; any Markdown other than headings for notes on how to read the file; then H2 sections, each a list of links with an optional description after a colon. A section titled "Optional" is understood, by convention, as material a tool may skip when its context is short. The example below is what this site's own file would look like, cut down.

$ curl -s https://example.net/llms.txt
# Example Net
> Vendor-neutral reference for load balancing, switching, routing and network
> monitoring, with an archive of a discontinued switch vendor. British spelling.

Guides state the standard they rely on and give configuration shapes as
generic examples. Archive pages describe historical products and are dated.

## Guides
- [sFlow](https://example.net/network-monitoring/sflow/index.html.md): sampling, counters, collector setup
- [Port mirroring](https://example.net/network-monitoring/port-mirroring/index.html.md): SPAN, oversubscription, TAP
- [VLANs](https://example.net/switching/vlans/index.html.md): tagging, trunks, inter-VLAN routing

## Optional
- [Company history](https://example.net/archive/company-history/index.html.md): 1996 to 2008 timeline
$ curl -sI https://example.net/network-monitoring/sflow/ | grep -i '^link'
link: </network-monitoring/sflow/index.html.md>; rel="alternate"; type="text/markdown", </llms.txt>; rel="describedby"

The second revision of the proposal, updated in 2026, added three things worth knowing. A file may sit at any path, not only the root, and then covers the pages beneath that path, so /docs/llms.txt describes /docs/ and a tool should prefer the most specific file. Pages may offer a clean Markdown copy at the same URL with .md appended, which is where the links in the file should point. And a page can announce both with standard link relations, rel="alternate" with type text/markdown for its own Markdown copy and rel="describedby" for the llms.txt that covers it, either as HTML link elements or as an HTTP Link header set at the proxy, which needs no change to the pages themselves.

06 /Serving both files from the edge

Both files are tiny and requested constantly, so serve them from cache with a short lifetime and a correct text/plain type. Keep them out of any bot-management rule that might return a challenge page instead of the file; a crawler that receives HTML where it expected robots.txt may treat the site as unrestricted or as blocked, depending on its parser. Log requests to both paths separately. A client that never fetches robots.txt but claims a crawler token is worth a second look, and the cryptographic bot verification methods now emerging give operators a stronger check than the file ever could.

For the enforcement half of the problem, the fingerprinting signals for automated clients and the balancer-side controls in an organisation-wide crawler policy pick up where these files stop.

07 /Failure modes, pitfalls and expert tips

• Links that redirect. A tool reading an llms.txt follows each link once. A link that returns 301 to a page that returns 301 is a wasted fetch, and some tools stop at the first redirect. Test every link with a HEAD request after each site change.
• Listing the whole site. A file with four hundred links is a sitemap in the wrong format. Keep the file small enough to sit in a tool's context; put the long tail in the Optional section or leave it out.
• Wrong content type. Both files must be served as text/plain or text/markdown. A proxy rule that returns an HTML challenge page for unknown clients breaks both at once, and the log shows a 200.
• Contradicting robots.txt. An llms.txt that points tools at paths robots.txt disallows for them is a trap for the compliant. Generate one from the other, or check them together in the build.
• Tip: log the two paths separately and treat a fetch of /llms.txt as the first signal that a language-model tool has visited; the visits that follow, when a tool cites a page and a reader clicks through, are counted under the referral traffic that answer engines send.
• Tip: cache both files at the edge with a short lifetime and a stale-while-revalidate window, the pattern described in cache lifetimes for the files crawlers fetch constantly, so an origin outage does not turn into a 5xx that pauses every compliant crawler.