Why does cosine similarity fail for semantic caching? Testing cosine, LLM prompts, and a purpose-built verifier: Jev

We tested cosine similarity, prompted LLMs, and a purpose-built verifier (Jev) on 500 labeled query pairs, and cosine was clearly the least accurate, serving near-misses like “capital of Kenya” for “capital of Jamaica.” Jev was comparable to strong small LLMs on accuracy at about half the latency and a similar or lower price, while a reasoning model was more accurate but about 22× slower.