A 440k-token benchmark went from 553 ms to 35 ms median time to first byte. We removed unnecessary prompt-cache routing work before the model call.
How we cut time to first byte by 94% for long prompts
calendar_today
October 3, 2026
domain
litellm