How AI is applied across API Evangelist and APIs.io. Read my AI disclosure →
API Evangelist API Evangelist
Discovery
Learnings
Guidance
Toolbox
Alignment
API Evangelist LLC

how many top sites block gptbot, claudebot and ccbot?

calendar_today October 30, 2026 domain crawlgraph

we read the robots.txt of the top 1,000 domains. about a quarter of readable files block the big training crawlers, and ccbot is blocked more often than gptbot.

open_in_new Read original post