How AI is applied across API Evangelist and APIs.io. Read my AI disclosure →
API Evangelist API Evangelist
Discovery
Learnings
Guidance
Toolbox
Alignment
API Evangelist LLC

AI Crawlers List: GPTBot, ClaudeBot, PerplexityBot and robots.txt Rules

calendar_today November 5, 2026 domain webcrawlerapi-com

Every AI crawler by vendor: user agents, IP lists and robots.txt rules to block training but stay in AI search, plus a news vs docs survey.

open_in_new Read original post