we read the robots.txt of the top 1,000 domains. about a quarter of readable files block the big training crawlers, and ccbot is blocked more often than gptbot.
how many top sites block gptbot, claudebot and ccbot?
calendar_today
October 30, 2026
domain
crawlgraph