How vLLM’s asynchronous KV offloading connector stores KV cache in CPU memory to reduce recomputation, improve throughput under memory pressure, and support pluggable offload backends.
Inside vLLM’s New KV Offloading Connector: Smarter Memory Transfer for Maximizing Inference Throughput
calendar_today
January 8, 2026
person
Or Ozeri, Danny Harnik (vLLM Team at IBM Research)
domain
vllm