Caching to Absorb Crawler Load: Keep Bots Off the Origin

01 /Why crawler traffic misses the cache

A cache in front of a web farm, whether a 2003 ServerIron with a caching module or a modern reverse proxy, earns its keep on popular pages. Crawlers do the opposite of popular: they walk archives, tag pages and old documentation that no person has opened this month. Every one of those requests is a cache miss that costs origin CPU and database time. The fix is a set of rules that make long-tail pages cacheable and let crawlers ask cheaply whether anything changed. This procedure sits in the load balancing section beside the balancer techniques it builds on.

02 /The procedure

1. 01

   Give every HTML page an explicit lifetime

   Send Cache-Control with a shared-cache lifetime (s-maxage) even for pages you consider dynamic. Ten minutes on a documentation page removes almost all repeat crawler hits; add stale-while-revalidate so the proxy can serve the old copy while it refreshes.

2. 02

   Answer conditional requests

   Emit a strong ETag or an accurate Last-Modified. A crawler that sends If-None-Match or If-Modified-Since then gets a 304 with no body, which costs a fraction of a full render. Make sure the proxy can answer these from cache without asking the origin.

3. 03

   Keep sitemap lastmod honest

   Well-behaved crawlers use the sitemap lastmod to decide what to refetch. If every URL carries the build date, they refetch everything; if each carries its real last change, they skip the rest.

4. 04

   Cache robots.txt, sitemaps and llms.txt

   These are the most requested files on a crawled site. Serve them from cache with a short lifetime and never let them fall through to a slow backend.

5. 05

   Normalise cache keys

   Strip tracking parameters and sort query strings in the cache key so the same page is not cached a dozen times under different URLs.

6. 06

   Rate-limit what is left

   Once caching absorbs the repeats, the remaining load is real novelty. Apply per-crawler limits to that, as in token-bucket rate limiting for crawlers.

03 /Worked example: headers that reduce crawler load on one page

Illustrative values. A documentation page at /docs/serveriron/slb-guide/ is rendered from a database on origin 10.3.0.11 behind a proxy on VIP 192.0.2.50. The origin sends Cache-Control: public, s-maxage=600, stale-while-revalidate=86400, a strong ETag computed from the body, and a Last-Modified equal to the last edit. The s-maxage directive is defined in RFC 9111 and stale-while-revalidate in RFC 5861. The proxy is told to answer conditional requests from its own copy and to coalesce simultaneous misses for one key into one origin fetch.

$ curl -s -D - -o /dev/null https://192.0.2.50/docs/serveriron/slb-guide/
HTTP/1.1 200 OK
Cache-Control: public, s-maxage=600, stale-while-revalidate=86400
ETag: "7d3f1a2c"
Last-Modified: Tue, 16 Jun 2026 09:12:40 GMT
X-Cache: MISS
$ curl -s -D - -o /dev/null https://192.0.2.50/docs/serveriron/slb-guide/ | grep -E 'X-Cache|Age'
Age: 3
X-Cache: HIT
$ curl -s -D - -o /dev/null -H 'If-None-Match: "7d3f1a2c"' https://192.0.2.50/docs/serveriron/slb-guide/
HTTP/1.1 304 Not Modified
ETag: "7d3f1a2c"
X-Cache: HIT

Follow the timers. The first request after the page changes is a miss and costs one render. For the next 600 seconds every request, from any client, is a hit, and conditional requests are answered 304 from the cached validator without the origin. From 600 to 87,000 seconds the copy is stale but still served, and the first request in that window triggers one background revalidation, which the origin answers 304 if nothing changed, so the copy is refreshed without a render. A crawler that visits this page once an hour therefore costs the origin at most one conditional request per hour, and usually nothing at all.

The same shape applies to images and other static objects, with one caution. A cache that re-encodes or optimises images on the way through can discard the embedded provenance those files carry. What a cache must not strip from a signed image is the list to check before enabling any transformation, and which metadata survives resizing tells you what the origin should embed in the first place.

04 /Cache rules by response type

05 /Measuring the effect

Split cache hit ratio by client class: verified crawlers, unverified crawler labels and everyone else. Before the change, crawler hit ratios on HTML are often under 20 percent while human traffic sits much higher. After it, the crawler figure should climb close to the human one, and the count of 304 responses to crawlers should rise. Watch origin CPU during a known crawl window rather than averages; the point is to flatten the peaks. The method for identifying crawler traffic gives you the client classes to split by.

06 /Pitfalls

• Personalised pages. Never cache pages that vary by login in a shared cache; mark them private and let crawlers see only the public version.
• Weak validators everywhere. An ETag that changes on every render (because it includes a timestamp) defeats 304s entirely.
• Session persistence on crawl traffic. Sticky sessions tie a crawler to one real server; for anonymous crawl traffic, spread requests with the methods in server load balancing across a pool instead.

Caching is the technical half of the answer. Which crawlers get the cached pages at all is a policy question, set out in writing an AI crawler access policy.

07 /Failure modes and pitfalls

• Set-Cookie on a cacheable page. A framework that starts a session on every request adds Set-Cookie to every response; a well-behaved shared cache refuses to store it, and the crawler hits the origin every time. Suppress the cookie until the user does something that needs one.
• Vary: User-Agent. One cache entry per user-agent string means one entry per crawler version and almost no hits. Vary only on Accept-Encoding and, when unavoidable, a normalised device class.
• Query strings in the key. Tracking parameters and reordered query strings multiply one page into dozens of keys. Normalise the key before lookup: strip known tracking parameters, sort the rest.
• Timestamps in the ETag. An ETag that includes the render time changes every render and never matches an If-None-Match. Hash the body, or a version number of the content.
• Revalidation storms. When a popular key expires, every concurrent miss goes to the origin unless the proxy coalesces them; on a crawl of paginated archives this happens for hundreds of keys within a minute. Enable request coalescing and stale-while-revalidate together.
• Caching the redirect but not the page. A cached 301 from an old URL with no lifetime on the target means the crawler pays two round trips and the origin still renders. Cache both ends.

08 /Expert tips

• Log the cache result and the client class on the same line, so hit ratio per class is a one-line query rather than a join.
• Run a crawl of your own sitemap from a second host once a week and read the origin request count during it; the number should approach the count of pages that changed that week.
• Keep stale-if-error on every public page so an origin outage is invisible to crawlers, which otherwise record errors against the site.
• Give the sitemap its own short lifetime and regenerate it on content change rather than on a timer, so lastmod is never older than the page it describes.

To reduce crawler load durably, make the cheap answer the default: a hit or a 304 for everything public, a coalesced single render for everything that changed, and transformation-free storage for anything that carries provenance. The measurements above are how you prove it worked, and the policy page decides who receives the cached pages at all.