Long-context workloads are quickly becoming the norm for enterprise AI. Agentic applications carry an ever-growing history of tool calls and intermediate results, retrieval-augmented generation prepends thousands of tokens of retrieved context, and multi-turn assistants resend the whole conversation with every turn. In all of these cases, the model sees large parts of the same prompt again and again.
Benchmarking tiered KV cache offloading on H100 GPUs in OpenShift AI 3.5
calendar_today
November 2, 2026
person
Ramesh Doddaiah, Alberto Perdomo, Nathan Scott, Ashish Kamra, Michey Mehta, Yuchen Fama, Naina Singh
domain
red-hat