How AI is applied across API Evangelist and APIs.io. Read my AI disclosure →
API Evangelist API Evangelist
Discovery
Learnings
Guidance
Toolbox
Alignment
API Evangelist LLC

LLM-as-a-judge vs human evaluation: how to check your judge

calendar_today November 11, 2026 person Rogerio Chaves domain langwatch

Check an LLM judge against human labels on real traces: annotation queues, agreement, Cohen’s kappa, a confusion table and the prompt fixes that closed the gap.

open_in_new Read original post