Up to 60% of GPU compute goes to recomputing context. Flash storage can put that capacity back to work

GPUs running large language model inference can spend up to 60% of their compute recalculating context they already processed, says Alan McSeveney, field CTO at WEKA. Speaking with VentureBeat lead analyst Rob Strechay at the latest stop in the VentureBeat AI Impact Series, McSeveney said the waste goes back to how large language models hold conversation history in GPU memory, arguing that fixing it lets companies serve several times more users on hardware they already own. The efficiencies WEKA customers struggle with most show up in infrastructure, separate from the application metrics each