KV Cache Routing: Why Standard Load Balancers Break Prefix Caching (and How to Fix It)

Standard round-robin load balancers cut KV cache hit rates in multi-replica LLM deployments, with benchmarks showing up to 57x faster time-to-first-token when switching to prefix-aware routing. The article outlines session affinity, prefix-hash routing, and KV-event-aware routing, which can boost throughput 2-3x on prefix-heavy workloads like RAG and multi-turn chat.