How AI is applied across API Evangelist and APIs.io. Read my AI disclosure →
API Evangelist API Evangelist
Discovery
Learnings
Guidance
Toolbox
Alignment
API Evangelist LLC

Scrolling via page API

calendar_today December 31, 0000 person domain scrapingbee

Some pages load more content only after you click “Load more results” or scroll and wait. In reality, the page often fetches additional results from its own API. If ScrapingBee can’t load those results, you can target the site’s API URL directly.

Here’s how to do that using this URL as an example: https://www.reuters.com/technology

*Note that the JavaScript Scenario has a maximum execution time limit of 40 seconds. Requests exceeding this limit will result in a timeout: https://www.scrapingbee.com/documentation/js-scenario/#timeout

open_in_new Read original post