Browser Run /crawl results now include more detailed diagnostics. Each URL record can include an outcome that identifies the specific processing result, such as success , robots_disallowed , timeout , or page_error . The job-level robots array reports the robots.txt state for up to 50 origins encountered by the crawl.
Browser Run - Crawl results include URL outcomes and robots.txt diagnostics
calendar_today
November 3, 2026
domain
cloudflare