How AI is applied across API Evangelist and APIs.io. Read my AI disclosure →
API Evangelist API Evangelist
Discovery
Learnings
Guidance
Toolbox
Alignment
API Evangelist LLC

Measuring agent consistency with pass^k

calendar_today November 13, 2026 domain launchdarkly

Use repeated eval runs, pass^k, and row-by-row comparisons to find which agent decisions got less reliable after a prompt or model change, even when accuracy goes up.

open_in_new Read original post