Cloudflare now scores verified bots on robots.txt and 429s, and a research crawler just ignored both. What the web measures, and a fetch loop that passes.
Need help?
Contact usCloudflare now scores verified bots on robots.txt and 429s, and a research crawler just ignored both. What the web measures, and a fetch loop that passes.