Routing LLM traffic across inference providers by cost, speed and reliability

Providers now sell the same open-weight models at very different prices and speeds. We describe an adaptive router that keeps us on the cheapest provider that is fast and healthy, and moves our traffic when that changes, without an engineer involved.