How AI is applied across API Evangelist and APIs.io. Read my AI disclosure →
API Evangelist API Evangelist
Discovery
Learnings
Guidance
Toolbox
Alignment
API Evangelist LLC

Push-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM

calendar_today October 14, 2026 person Sunita Nadampalli (AWS), Nicolò Lucchesi (Mistral) domain vllm

A push-based NIXL connector for vLLM disaggregated serving: the prefill instance writes KV directly into decode memory over RDMA, taking the router relay, decode-side allocation, and transfer setup off the TTFT critical path.

open_in_new Read original post