Grading the Machine: Using LLM as a Judge to Monitor AI Agents in Production

State Farm engineering explores using large language models as evaluators of AI chatbot quality in production, using reference-free metrics to surface issues without pre-written correct answers. The piece covers building custom evaluation criteria, meta-analysis of failure patterns, and validating the judge’s reliability against human reviewers using Claude Sonnet 4.5 and AWS services.