Docker Containers Don’t Know Your Model Is Still Loading

It was a Friday at 4:50 pm, the worst possible time for anything to go sideways when marketing flipped on a new AI summarization feature for the whole user base instead of the 5% rollout we’d agreed on. Traffic to our LLM service doubled in about four minutes. The autoscaler did exactly what it was told: it spun up three new replicas.