llm-d - Distributed Inference Serving on Kubernetes

Today, Red Hat, Google, and IBM announced an exciting new open-source project called llm-d ; a distributed inference platform built around vLLM . Personally, I’m very excited about this project as we know working with users and the community how difficult it is to build a cost-effective and performant inference platform. Organizations trying to run inference themselves have likely run into the main motivation for the llm-d project ( quoted from the press release ): The escalating resource demands of increasingly sophisticated and larger reasoning models limits the viability of centralized…