The LLM Evaluation Score Improved. Production Quality Didn't.

An LLM application can post a higher evaluation score and still become less useful in production. The contradiction is only apparent. An evaluation score is not quality itself, but it is an estimate produced by a particular dataset, metric, judge, sampling strategy, and aggregation rule.