Every AI crawler by vendor: user agents, IP lists and robots.txt rules to block training but stay in AI search, plus a news vs docs survey.
AI Crawlers List: GPTBot, ClaudeBot, PerplexityBot and robots.txt Rules
calendar_today
November 5, 2026
domain
webcrawlerapi-com