Crawler Rate Limiting on a Load Balancer or Reverse Proxy

01 /Why crawlers need their own limits

A crawler differs from a user in the way that matters to capacity: it walks the long tail. People concentrate on a few popular pages that stay in cache. A crawler requests every archive page, tag listing and paginated view, most of them cold, so each request lands on the origin. Ten requests a second from one crawler can cost more origin CPU than a thousand users. This procedure belongs to the load balancing hub and assumes you already know which crawlers you accept, from measuring and verifying crawler traffic.

02 /Token bucket and leaky bucket

A token bucket holds up to B tokens and refills at R tokens per second. Each request spends one; with the bucket empty, the request is refused. It allows a burst of B, then a sustained rate of R, which suits crawlers that fetch a sitemap and then pause. A leaky bucket queues requests and releases them at a fixed rate R; the queue depth is the burst allowance. It smooths traffic into the origin but adds latency to queued requests. Many proxy implementations use leaky bucket semantics with an optional burst that is served immediately rather than delayed, which behaves much like a token bucket.

03 /Procedure

1. 01

   Cache before you limit

   Set cache lifetimes on the balancer or proxy for pages that tolerate them, and serve stale content while revalidating. A cache hit costs the origin nothing, and most crawler load vanishes once deep pages are cacheable for even a few minutes.

2. 02

   Classify the client

   Tag each request as verified crawler, claimed but unverified crawler, suspected bot or other. Use DNS or published-range verification for the first, and fingerprint and behaviour scoring for the third. The class selects the bucket.

3. 03

   Choose the key and bucket

   Key verified crawlers on their identity, other clients on address or prefix. Size rate R from what the origin can serve uncached, and burst B from what a well-behaved crawler does after reading a sitemap.

4. 04

   Refuse correctly

   Return 429 Too Many Requests with a Retry-After value in seconds. Compliant crawlers back off. Keep 503 for genuine origin overload, because some crawlers read repeated 503s as the site being down and change how they schedule it.

5. 05

   Protect health checks and the origin

   Exempt the balancer's own health checks and your monitoring addresses from every limit, and apply limits in front of the real server pool, never on the path the balancer uses to test it. A limiter that fails health checks takes servers out of rotation and turns a crawler problem into an outage.

6. 06

   Log and review

   Log key, class, rule, bucket level and decision for every refused request and a sample of passed ones. Review weekly: rising 429s to a verified crawler mean the limit is too tight or the cache too cold.

04 /Illustrative config shape

The shape below uses generic open-source reverse proxy syntax with leaky bucket semantics. A map chooses the key: verified crawlers share a key per identity, everyone else is keyed on address. It is a sketch, not a drop-in configuration.

map $crawler_class $limit_key {
    verified    $crawler_name;
    default     $binary_remote_addr;
}
limit_req_zone $limit_key zone=crawl:10m rate=5r/s;
limit_req_status 429;
server {
    listen 443 ssl;
    server_name www.example.com;
    location = /healthz { limit_req off; }
    location / {
        limit_req zone=crawl burst=20 nodelay;
        error_page 429 = @throttled;
        # hand off to the real server pool here
    }
    location @throttled {
        add_header Retry-After 30 always;
        return 429;
    }
}

05 /Where the limiter sits

Put it at the first device that terminates HTTP, usually a Layer 7 balancer or reverse proxy, because it needs the user-agent and path. A Layer 4 device can only limit connections per address, which is blunt. Keep limiter state local to each balancer unless the pair shares it; two units each allowing 5 requests a second allow 10 in total. That matters under persistence rules that pin a crawler to one unit, and it matters in reverse when they do not. Watch the effect on volume by prefix with sampled flow data after each change.

Rate limits are the last line. Before them, caching rules that absorb crawler requests remove most repeat load, and a written AI crawler policy decides who is limited at all.