Writing an AI Crawler Access Policy for Your Organisation

01 /Why write the policy down

The engineers who configure balancers are usually the people asked to "block the AI bots" and later asked why the site stopped appearing in answers. A written policy separates the decision from the configuration: someone accountable decides what the organisation wants, and operations implements it. It also stops drift, because every rule on the edge can be traced to a line in the document. This page belongs to the load balancing section because the balancer is where most of the policy ends up enforced.

02 /Decide per operator and purpose

Treat each crawler operator as up to three clients, because their purposes differ. A training crawler collects pages for model building. An answer fetcher retrieves pages at question time and may cite them. An agent acts on a user's behalf. An organisation may reasonably accept the second, which can send readers, while declining the first. The table below is the core of the policy document.

03 /Expressing the policy

The policy lives in two places. For compliant crawlers, robots.txt groups and control tokens state the rules, and an llms.txt file can point answer tools at the pages you most want read. For everything else, the rules live at the edge: address sets or signature checks to identify the operator, then allow, rate-limit or deny. A rule that exists only in robots.txt is a request; a rule that exists only at the edge is invisible to honest crawlers, which then keep trying. Put every decision in both places.

04 /Identity before policy

No rule should key on the user-agent string alone. Verify the operator first, using signed requests where available and forward-confirmed DNS or published ranges otherwise, then apply that operator's line of the policy. Anything claiming an operator's identity that fails verification gets the "unverified" line, which is usually a block. Shaping itself is a balancer job: rate limiting crawlers with token buckets and 429 responses gives the configuration, and absorbing crawler load with caching reduces how much shaping you need.

05 /Reviewing it

1. 01

   Measure

   Each quarter, pull requests, bytes and origin time by verified operator and purpose from the balancer logs.

2. 02

   Compare

   Check measured behaviour against the policy: operators honouring Disallow, fetchers staying inside their rate, no growth in the unverified bucket.

3. 03

   Decide

   Bring changes to the owner: new operators to classify, rates to adjust, decisions to revisit as the organisation's view changes.

4. 04

   Implement and record

   Change robots.txt and edge rules together, and add the date and approver to the table.