Load Balancing LLM Inference Servers: What Changes

01 /Why inference traffic is different

The load balancing methods in this load balancing section were designed for web traffic: short requests of roughly equal cost, where spreading connections evenly spreads work evenly. Language model inference breaks both assumptions. A request holds a connection open while the model generates, often streaming tokens for tens of seconds. Its cost depends on the length of the prompt and of the answer, which can differ by a factor of a hundred between two requests. And a server's capacity is set by accelerator memory, not by connection count.

02 /Which methods still work

Round-robin fails first: it sends the next request to the next server regardless of whether that server is halfway through three long generations. Least-connections, the default on many balancers since the classic server load balancing designs, is better but still counts a one-sentence request the same as a long document summary. What works is routing on signals the inference server exposes: the number of requests queued, the memory free for per-request state, and recent time to first token. Several open-source inference servers publish these as metrics, and newer gateway projects route on them directly.

03 /Prefix affinity: the new persistence

Inference servers cache the computed state of a prompt (often called the KV cache). When many requests share a long prefix, such as the same system instructions or the same document, sending them to the server that already holds that prefix skips a large part of the work. That is a new kind of affinity, closer to cache locality than to the cookie-based session persistence methods of the web era, and it has the same trade-off: stick too hard and one server overheats. Good gateways hash on the prefix but spill to another server when the preferred one's queue grows.

04 /LLM load balancing worked example: a four-server pool

Illustrative values. Four inference servers, 10.4.0.11 to 10.4.0.14, each hold one copy of the same model and expose an HTTP API on 8000 and a metrics endpoint on 8001. The gateway on VIP 192.0.2.60 scrapes metrics every 2 seconds and keeps, per server, the number of requests running, the number waiting, the fraction of accelerator memory used for cached prompt state, and the last observed time to first token. Routing is prefix hash first, then a queue-depth check, then least waiting requests as the tie-breaker. The idle timeout on the client leg is 600 seconds and buffering is off.

gw1# show inference pool chat-pool
server        running  waiting  kv-mem  ttft-ms  state
10.4.0.11     14       0        61%     240      UP
10.4.0.12     16       3        88%     910      UP
10.4.0.13     9        0        44%     180      UP
10.4.0.14     0        0        3%      -        DRAINING
$ curl -s http://10.4.0.12:8001/metrics | grep -E 'requests_(running|waiting)'
inference_requests_running 16
inference_requests_waiting 3
$ curl -sN -H 'Content-Type: application/json' -d @prompt.json https://192.0.2.60/v1/completions | head -3
data: {"text": "Server"}
data: {"text": " load"}
data: {"text": " balancing"}

Walk one request through. Its prompt begins with the same 2,000 token system preamble as most traffic, and the hash of that prefix points at 10.4.0.12. That server has 3 requests waiting and a time to first token near a second, above the spill threshold of 500 milliseconds, so the gateway abandons affinity for this request and sends it to 10.4.0.13, which has no queue and the lowest observed latency. 10.4.0.13 recomputes the prefix once and then holds it, so the next request with the same prefix has two warm candidates. 10.4.0.14 is draining for an update: it takes nothing new and will be removed when running reaches zero.

The stream in the transcript is server-sent events, the text/event-stream format defined in the WHATWG HTML standard, which is why the client uses -N to disable its own buffering and why the gateway must do the same. A proxy that assembles the whole body would deliver every token at once, after the model finished.

05 /Timeouts and limits to change for an inference pool

06 /Streaming and timeouts

Streamed answers use long-lived HTTP responses, server-sent events or WebSockets. Balancer idle timeouts tuned for web traffic, often 60 seconds, cut long generations off mid-answer. Raise them for the inference pool, disable response buffering so tokens reach the client as they are produced, and make sure a Layer 7 device is not waiting for the full body before forwarding. The difference between terminating at Layer 4 and Layer 7 matters here, as covered in what a Layer 7 balancer sees and changes.

07 /Health checks that test the model

A TCP connect or an HTTP 200 on a status path says the process is up, not that the model can generate. Use an application health check that asks for a tiny completion with a short timeout, and take a server out of rotation when it fails or when its queue passes a threshold. Drain rather than cut: let in-flight streams finish before removing a server, or users see answers stop halfway.

08 /Failure modes and pitfalls

• Affinity without spill. A prefix hash with no queue check sends every request sharing a popular preamble to one server until it stalls, while the rest of the pool idles. Set a spill threshold on waiting requests or time to first token and test it by flooding one prefix.
• Retrying a partial stream. A gateway that retries on server error after the first tokens have been sent gives the client a second copy of the beginning of the answer. Retries are safe only before the first byte.
• Stale metrics. A scrape interval of 15 seconds against requests that arrive in bursts routes on a picture that is already wrong. Scrape every second or two, or have the server report queue depth in a response header the gateway can read on every reply.
• Health checks that spend capacity. A completion check on a saturated server queues behind real requests, times out and removes the one server that was doing the most work. Give the check its own short queue or a reserved slot, and require repeated failures.
• HTTP/2 flow control on a slow client. One client that reads slowly can hold a stream's window shut while the server's output buffer fills. Set per-stream limits at the gateway so one slow reader does not stall a server's other streams.
• Agents that pipeline many calls. Automated clients that fire dozens of dependent completions in sequence hold slots far longer than an interactive user; the connection patterns agent clients produce are worth a separate per-client concurrency limit.

09 /Expert tips

• Measure time to first token and tokens per second at the gateway, per server, and alert on the first; it rises before queue depth does.
• Limit by tokens in flight rather than requests where the server exposes it; ten short questions and one long document are not the same load.
• Put a client certificate on the leg from gateway to model servers, because the model API has no session or user concept of its own and will answer anyone who can reach port 8000.
• Drain a server by weight zero and wait for running to reach zero before any update; a hard restart mid-stream is visible to every user on that server at once.

LLM load balancing keeps the shape of server load balancing, a VIP in front of a pool with health checks and a selection method, and changes the numbers inside it: seconds become minutes and cookies become prefixes. Get the timers and the spill rule right first; the rest is ordinary balancer discipline.