Inference latency: what it measures & why it varies

Ask an engineer what their LLM app’s inference latency is, and the honest answer is “which one?” The time to the first visible token, the time to the finished response, and the time an agent spends across a chain of calls are three different numbers. …