Caching to Absorb Crawler Load: Keep Bots Off the Origin

01 /Why crawler traffic misses the cache

A cache in front of a web farm, whether a 2003 ServerIron with a caching module or a modern reverse proxy, earns its keep on popular pages. Crawlers do the opposite of popular: they walk archives, tag pages and old documentation that no person has opened this month. Every one of those requests is a cache miss that costs origin CPU and database time. The fix is not a bigger cache but rules that make long-tail pages cacheable and let crawlers ask cheaply whether anything changed. This procedure sits in the load balancing section beside the balancer techniques it builds on.

02 /The procedure

1. 01

   Give every HTML page an explicit lifetime

   Send Cache-Control with a shared-cache lifetime (s-maxage) even for pages you consider dynamic. Ten minutes on a documentation page removes almost all repeat crawler hits; add stale-while-revalidate so the proxy can serve the old copy while it refreshes.

2. 02

   Answer conditional requests

   Emit a strong ETag or an accurate Last-Modified. A crawler that sends If-None-Match or If-Modified-Since then gets a 304 with no body, which costs a fraction of a full render. Make sure the proxy can answer these from cache without asking the origin.

3. 03

   Keep sitemap lastmod honest

   Well-behaved crawlers use the sitemap lastmod to decide what to refetch. If every URL carries the build date, they refetch everything; if each carries its real last change, they skip the rest.

4. 04

   Cache robots.txt, sitemaps and llms.txt

   These are the most requested files on a crawled site. Serve them from cache with a short lifetime and never let them fall through to a slow backend.

5. 05

   Normalise cache keys

   Strip tracking parameters and sort query strings in the cache key so the same page is not cached a dozen times under different URLs.

6. 06

   Rate-limit what is left

   Once caching absorbs the repeats, the remaining load is real novelty. Apply per-crawler limits to that, as in token-bucket rate limiting for crawlers.

03 /Measuring the effect

Split cache hit ratio by client class: verified crawlers, unverified crawler labels and everyone else. Before the change, crawler hit ratios on HTML are often under 20 percent while human traffic sits much higher. After it, the crawler figure should climb close to the human one, and the count of 304 responses to crawlers should rise. Watch origin CPU during a known crawl window rather than averages; the point is to flatten the peaks. The method for identifying crawler traffic gives you the client classes to split by.

04 /Pitfalls

• Personalised pages. Never cache pages that vary by login in a shared cache; mark them private and let crawlers see only the public version.
• Weak validators everywhere. An ETag that changes on every render (because it includes a timestamp) defeats 304s entirely.
• Session persistence on crawl traffic. Sticky sessions tie a crawler to one real server; for anonymous crawl traffic, spread requests with the methods in server load balancing across a pool instead.

Caching is the technical half of the answer. Which crawlers get the cached pages at all is a policy question, set out in writing an AI crawler access policy.