Jev vs. LLM-as-a-Judge for AI Evals

A model that answers yes/no questions instead of writing text came out 3.5× cheaper and 3.8× faster than our gpt-4o-mini judge on 1,000 real production turns, and agreed with it on 897 of them. Here’s the test that showed it, what the 103 disagreements have in common, and how to run it yourself the next […] The post Jev vs. LLM-as-a-Judge for AI Evals appeared first on Comet .