Does quantization speed up inference?

Running a large language model isn’t cheap. Every response burns GPU time, memory, and money, and those costs grow as your app grows. Quantization is one of the most common tricks for making models cheaper and faster to run, which is why you’ll see it…