How simulations reduce the guesswork in our infrastructure decisions

We had a standby pool algorithm with over a dozen tunable parameters and no good way to test changes without risking real customer latency. So we built a simulator fed with real job data, ran hyperparameter optimization on it, and found a configuration that was faster and cheaper.