Writing an AI Crawler Access Policy for Your Organisation

01 /Why write the policy down

The engineers who configure balancers are usually the people asked to "block the AI bots" and later asked why the site stopped appearing in answers. A written policy separates the decision from the configuration: someone accountable decides what the organisation wants, and operations implements it. It also stops drift, because every rule on the edge can be traced to a line in the document. This page belongs to the load balancing section because the balancer is where most of the policy ends up enforced.

02 /Decide per operator and purpose

Treat each crawler operator as up to three clients, because their purposes differ. A training crawler collects pages for model building. An answer fetcher retrieves pages at question time and may cite them. An agent acts on a user's behalf. An organisation may reasonably accept the second, which can send readers, while declining the first. The table below is the core of the policy document.

03 /Expressing the policy

The policy lives in two places. For compliant crawlers, robots.txt groups and control tokens state the rules, and an llms.txt file can point answer tools at the pages you most want read. For everything else, the rules live at the edge: address sets or signature checks to identify the operator, then allow, rate-limit or deny. A rule that exists only in robots.txt is a request; a rule that exists only at the edge is invisible to honest crawlers, which then keep trying. Put every decision in both places.

04 /Identity before policy

No rule should key on the user-agent string alone. Verify the operator first, using signed requests where available and forward-confirmed DNS or published ranges otherwise, then apply that operator's line of the policy. Anything claiming an operator's identity that fails verification gets the "unverified" line, which is usually a block. Shaping itself is a balancer job: rate limiting crawlers with token buckets and 429 responses gives the configuration, and absorbing crawler load with caching reduces how much shaping you need.

05 /One AI crawler policy line, enforced end to end

Take the second row of the table: answer fetcher, operator A, allowed and shaped at 2 requests per second per verified identity, approved by the web lead. Illustrative values follow. The operator publishes its fetcher's address ranges and signs its requests; the site's edge is a Layer 7 balancer on VIP 192.0.2.70. The line becomes three artefacts: a robots.txt group, an edge rule, and a log line format that the quarterly review can query.

$ curl -s https://www.example.com/robots.txt | sed -n '/operator-a-fetcher/,/^$/p'
User-agent: operator-a-fetcher
Allow: /
Disallow: /account/
# policy row 2, approved web lead 2026-09, review 2026-12
lb1# show policy crawler operator-a-fetcher
identity   operator-a-fetcher   verify: signature, then published-ranges
verified   allow  rate 2/s burst 10  log class=fetcher-a
unverified deny   log class=impersonation
$ grep 'class=fetcher-a' /var/log/edge/access.log | tail -1
10:41:07 client=198.51.100.23 class=fetcher-a verify=sig rule=policy-2 decision=allow tokens=9 path=/docs/bigiron/ status=304
$ grep 'class=impersonation' /var/log/edge/access.log | tail -1
10:41:09 client=203.0.113.77 class=impersonation verify=fail rule=policy-4 decision=deny path=/docs/bigiron/ status=403

The comment in robots.txt and the rule name at the edge both carry the row number and the review date, so a reader of either artefact can find the decision that produced it. The two log lines are the evidence the review needs: the first shows a verified fetcher inside its budget and receiving a cheap 304, the second shows a client borrowing the fetcher's name and failing the identity check that decides which policy line applies. When the operator changes its ranges or its signing key, only the verification input changes; the policy row and the rule name do not.

Note what the policy line did not do. It did not block the operator's training crawler, which is a separate row with a separate decision, and it did not block the operator's agents, which act for signed-in users and are governed by an application rule. One operator, three rows.

06 /Evidence sources for the quarterly review

Because the review may lead to blocking an operator, or to a dispute with one, the access log has to be more than a text file someone could have edited. Retained edge logs with tamper evidence are what make the quarterly numbers defensible.

07 /Reviewing it

1. 01

   Measure

   Each quarter, pull requests, bytes and origin time by verified operator and purpose from the balancer logs.

2. 02

   Compare

   Check measured behaviour against the policy: operators honouring Disallow, fetchers staying inside their rate, no growth in the unverified bucket.

3. 03

   Decide

   Bring changes to the owner: new operators to classify, rates to adjust, decisions to revisit as the organisation's view changes.

4. 04

   Implement and record

   Change robots.txt and edge rules together, and add the date and approver to the table.

08 /Failure modes and pitfalls

• Robots.txt and the edge disagree. The document says Disallow, the balancer says allow, or the reverse. Compliant crawlers follow the file, the edge enforces something else, and the review finds traffic the policy never approved. Change both in one ticket, or generate one from the other.
• Rules keyed on user-agent alone. Any client can claim any name. Without verification, a block on a training crawler blocks nothing that wanted to get in, and a generous allowance for a fetcher is available to anyone who types its name.
• Blocking by network instead of by operator. A block on a hosting provider's whole address space to stop one crawler also blocks every other customer of that provider, including monitoring services and partners.
• Decisions with no expiry. A row approved two years ago by someone who has left is still enforced. Every row carries a review date, and a row past its date is flagged, not silently kept.
• Blocking the fetcher that sends readers. The organisation wanted to keep its pages out of training sets and instead removed itself from answers that cite sources. Separate the purposes before deciding.
• Cache in front of the policy. A proxy that serves cached pages before the policy rule runs hands blocked operators every popular page. Verify and classify before the cache lookup, even when the answer is a hit.

09 /Expert tips

• Number the rows and put the number in every artefact: the robots.txt comment, the edge rule name, the log field. Traceability is what makes the policy auditable.
• Add a standing row for operators not yet classified: log and allow at a low rate for one cycle, then bring them to the owner. New operators appear between reviews.
• Keep the policy table in a repository with history, and the edge configuration next to it, so a diff shows who changed which row and when.
• Give the policy owner a one-page quarterly report drawn from the evidence table, in the same row order as the policy, so decisions and outcomes line up.

An AI crawler policy earns its keep when every edge rule can be traced to a row, every row to a person and a date, and every quarterly number to a log that could not have been quietly changed. Write the table, wire the identity check in front of it, and let the balancer do the rest.