P99 latency: What it means, why it matters & how to fix it in LLM apps

Your LLM app’s average response time looks great, but users are still complaining. That disconnect comes down to math. Average latency can mask how bad the slowest requests are, while p99 latency shows you what the worst 1% of requests look like.