High Performance Distributed Inference with Ray Serve LLM

Anyscale announces major performance improvements to Ray Serve LLM through three optimizations: direct streaming, a new vLLM Ray executor backend, and HAProxy integration. The changes achieve up to 4.4x higher throughput on prefill-heavy workloads and up to 24x on decode-heavy workloads, now matching vllm-router performance, demonstrated on vLLM and Google Kubernetes Engine (GKE).