On this page
01 Why inference traffic is different
The load balancing methods in this load balancing section were designed for web traffic: short requests of roughly equal cost, where spreading connections evenly spreads work evenly. Language model inference breaks both assumptions. A request holds a connection open while the model generates, often streaming tokens for tens of seconds. Its cost depends on the length of the prompt and of the answer, which can differ by a factor of a hundred between two requests. And a server's capacity is set by accelerator memory, not by connection count.
| Property | Web request | Inference request |
|---|---|---|
| Duration | Milliseconds | Seconds to minutes |
| Response | One body | A stream of tokens |
| Cost per request | Roughly uniform | Varies by prompt and output length |
| Capacity limit | CPU and connections | Accelerator memory for model state |
| Useful affinity | Session state (rarely needed) | Cached prompt prefix on one server |
02 Which methods still work
Round-robin fails first: it sends the next request to the next server regardless of whether that server is halfway through three long generations. Least-connections, the default on many balancers since the classic server load balancing designs, is better but still counts a one-sentence request the same as a long document summary. What works is routing on signals the inference server exposes: the number of requests queued, the memory free for per-request state, and recent time to first token. Several open-source inference servers publish these as metrics, and newer gateway projects route on them directly.
03 Prefix affinity: the new persistence
Inference servers cache the computed state of a prompt (often called the KV cache). When many requests share a long prefix, such as the same system instructions or the same document, sending them to the server that already holds that prefix skips a large part of the work. That is a new kind of affinity, closer to cache locality than to the cookie-based session persistence methods of the web era, and it has the same trade-off: stick too hard and one server overheats. Good gateways hash on the prefix but spill to another server when the preferred one's queue grows.
04 Streaming and timeouts
Streamed answers use long-lived HTTP responses, server-sent events or WebSockets. Balancer idle timeouts tuned for web traffic, often 60 seconds, cut long generations off mid-answer. Raise them for the inference pool, disable response buffering so tokens reach the client as they are produced, and make sure a Layer 7 device is not waiting for the full body before forwarding. The difference between terminating at Layer 4 and Layer 7 matters here, as covered in what a Layer 7 balancer sees and changes.
05 Health checks that test the model
A TCP connect or an HTTP 200 on a status path says the process is up, not that the model can generate. Use an application health check that asks for a tiny completion with a short timeout, and take a server out of rotation when it fails or when its queue passes a threshold. Drain rather than cut: let in-flight streams finish before removing a server, or users see answers stop halfway.
Measure time to first token at the balancer. It is the number users feel, and it rises before anything else when a pool is overloaded.
06 Questions
What is the best load balancing algorithm for LLM inference?
One that routes on the inference server's real load, such as queue depth and free memory for request state, with prefix affinity for shared prompts. Plain round-robin and least-connections spread connections, not work.
Why do long AI responses get cut off behind a load balancer?
Usually an idle or response timeout tuned for short web requests, or response buffering at Layer 7. Raise timeouts for the inference pool and stream without buffering.
What is prefix-aware routing?
Sending requests that share a prompt prefix to the server that already cached that prefix, so it does not recompute it. It is a form of affinity with spill-over when the preferred server is busy.
How should I health-check an inference server?
Ask it for a very short generation with a tight timeout, and also watch its queue depth. A port check alone misses a model that is loaded but stuck.
Does session persistence still matter?
Cookie persistence rarely does for inference APIs. Prefix affinity has taken its place.