Speculative decoding: How it works, when it helps & where it fits in your inference stack

You’re running LLM inference in production. Semantic caching handles the easy wins: repeated queries with the same intent come back from cache without touching the model. But everything else still hits the model at full cost, and that adds up fast at …