Crawler Rate Limiting on a Load Balancer or Reverse Proxy

01 /Why crawlers need their own limits

A crawler differs from a user in the way that matters to capacity: it walks the long tail. People concentrate on a few popular pages that stay in cache. A crawler requests every archive page, tag listing and paginated view, most of them cold, so each request lands on the origin. Ten requests a second from one crawler can cost more origin CPU than a thousand users. This procedure belongs to the load balancing hub and assumes you already know which crawlers you accept, from measuring and verifying crawler traffic.

02 /Token bucket and leaky bucket

A token bucket holds up to B tokens and refills at R tokens per second. Each request spends one; with the bucket empty, the request is refused. It allows a burst of B, then a sustained rate of R, which suits crawlers that fetch a sitemap and then pause. A leaky bucket queues requests and releases them at a fixed rate R; the queue depth is the burst allowance. It smooths traffic into the origin but adds latency to queued requests. Many proxy implementations use leaky bucket semantics with an optional burst that is served immediately rather than delayed, which behaves much like a token bucket.

03 /Procedure

1. 01

   Cache before you limit

   Set cache lifetimes on the balancer or proxy for pages that tolerate them, and serve stale content while revalidating. A cache hit costs the origin nothing, and most crawler load vanishes once deep pages are cacheable for even a few minutes.

2. 02

   Classify the client

   Tag each request as verified crawler, claimed but unverified crawler, suspected bot or other. Use DNS or published-range verification for the first, and fingerprint and behaviour scoring for the third. The class selects the bucket.

3. 03

   Choose the key and bucket

   Key verified crawlers on their identity, other clients on address or prefix. Size rate R from what the origin can serve uncached, and burst B from what a well-behaved crawler does after reading a sitemap.

4. 04

   Refuse correctly

   Return 429 Too Many Requests with a Retry-After value in seconds. Compliant crawlers back off. Keep 503 for genuine origin overload, because some crawlers read repeated 503s as the site being down and change how they schedule it.

5. 05

   Protect health checks and the origin

   Exempt the balancer's own health checks and your monitoring addresses from every limit, and apply limits in front of the real server pool, never on the path the balancer uses to test it. A limiter that fails health checks takes servers out of rotation and turns a crawler problem into an outage.

6. 06

   Log and review

   Log key, class, rule, bucket level and decision for every refused request and a sample of passed ones. Review weekly: rising 429s to a verified crawler mean the limit is too tight or the cache too cold.

04 /Illustrative config shape

The shape below uses generic open-source reverse proxy syntax with leaky bucket semantics. A map chooses the key: verified crawlers share a key per identity, everyone else is keyed on address. It is a sketch, not a drop-in configuration.

map $crawler_class $limit_key {
    verified    $crawler_name;
    default     $binary_remote_addr;
}
limit_req_zone $limit_key zone=crawl:10m rate=5r/s;
limit_req_status 429;
server {
    listen 443 ssl;
    server_name www.example.com;
    location = /healthz { limit_req off; }
    location / {
        limit_req zone=crawl burst=20 nodelay;
        error_page 429 = @throttled;
        # hand off to the real server pool here
    }
    location @throttled {
        add_header Retry-After 30 always;
        return 429;
    }
}

05 /Rate limiting crawlers: a worked example with timers

Illustrative values. The origin pool behind VIP 192.0.2.40 can render uncached pages at a rate the operators measured during a quiet hour; the budget they are willing to give crawlers is a quarter of it. A verified crawler named crawler-a is keyed on its identity across all its addresses with a sustained rate of 5 requests per second and a burst of 20. Unverified clients that present a crawler user-agent are keyed per address at 1 request per second, burst 5. Retry-After is 30 seconds for the first class and 120 for the second.

Verification comes before the key. A request claiming crawler-a whose source fails the reverse DNS, published range or signed-request check falls into the unverified class, limited per address, and never inherits the named crawler's larger bucket.

$ curl -s -D - -o /dev/null -A 'crawler-a/1.0' https://192.0.2.40/archive/2009/page/412
HTTP/1.1 429 Too Many Requests
Retry-After: 30
Cache-Control: no-store
Content-Length: 0
$ tail -1 /var/log/proxy/limit.log
10:22:41 key=crawler-a class=verified rule=crawl rate=5 burst=20 tokens=0 decision=429 retry=30 path=/archive/2009/page/412
$ tail -1 /var/log/proxy/limit.log   # an unverified client using the same user-agent
10:22:43 key=203.0.113.9 class=unverified rule=crawl-unverified rate=1 burst=5 tokens=0 decision=429 retry=120 path=/tag/foundry

Read the bucket arithmetic. A crawler that fires 60 requests in one second after reading the sitemap spends its 20 burst tokens, receives 40 refusals with a 30 second wait and, if it obeys, returns to a bucket refilled to 20 (5 tokens per second for 30 seconds, capped at the bucket size) and settles into 5 per second. The same crawler at a steady 4 per second never sees a 429.

06 /Status codes for excess requests compared

07 /Where the limiter sits

Put it at the first device that terminates HTTP, usually a Layer 7 balancer or reverse proxy, because it needs the user-agent and path. A Layer 4 device can only limit connections per address, which is blunt. Keep limiter state local to each balancer unless the pair shares it; two units each allowing 5 requests a second allow 10 in total. That matters under persistence rules that pin a crawler to one unit, and it matters in reverse when they do not. Watch the effect on volume by prefix with sampled flow data after each change.

08 /Failure modes and pitfalls

• Keying on a spoofable header. A limiter that keys on X-Forwarded-For without stripping client-supplied values lets any client pick a fresh key per request. Key on the address the balancer itself observed, or on the verified identity.
• Counting 304s as full renders. A conditional request the cache answers with 304 costs the origin nothing; charging it a full token throttles the best-behaved crawlers first.
• Per-address keys on IPv6. A single host can present a new /128 per request from one /64. Key IPv6 clients on the /64 at least, or on the /48 for hosting ranges.
• Logs that cannot be audited later. When an operator disputes a limit, the log line is the only evidence; keep key, class and bucket state in retained edge logs that can be shown to have been unaltered.

09 /Expert tips

• Start with a bucket you believe is too generous and tighten from the 429 count; a limit that is too tight on day one is discovered by the crawler dropping your pages.
• Publish the intended rate for verified crawlers in the policy document and keep the edge configuration in step with it, so operations and the policy owner argue from the same number.
• Order the pipeline verify, cache, then limit; a limiter in front of the cache refuses requests the cache would have served for free.

Rate limiting crawlers is the last of three controls, after caching and policy, and it only works honestly when identity is checked first and every refusal is written down.

Rate limits are the last line. Before them, caching rules that absorb crawler requests remove most repeat load, and a written AI crawler policy decides who is limited at all.