Load Balancing AI Services for Availability and Speed

How Pinecone built a service-aware load balancer for Pinecone Assistant that routes across embeddings, rerankers, and LLMs in real time. Covers the tradeoffs of common load balancing strategies — least loaded, latency-aware, consistent hashing, round-robin — and explains why the power of two choices algorithm struck the right balance between simplicity and adaptability. Includes production results across six weeks: latency improvements on embeddings and rerankers, automatic failover during four upstream incidents, and a near-elimination of manual routing interventions.