How AI is applied across API Evangelist and APIs.io. Read my AI disclosure →
API Evangelist API Evangelist
Discovery
Learnings
Guidance
Toolbox
Alignment
API Evangelist LLC

The Web Has Started Grading Crawlers

calendar_today October 30, 2026 person Viktor Petrov domain crawlbase

Cloudflare now scores verified bots on robots.txt and 429s, and a research crawler just ignored both. What the web measures, and a fetch loop that passes.

open_in_new Read original post