I’m prototyping a small-model inference setup on an RTX 5070 (12GB VRAM) and running into the classic problem: KV cache grows linearly with sequence length, leaving little room for actual model weights. For context: I’m working with Grouped Query Attention (GQA) models and trying to combine PackKV-style token-wise quantization (4-bit) with DynamicKV’s per-layer retention budgets. This gets me ~75% memory reduction on random tensors, but I haven’t tested on real model outputs yet.