Parsing TDMRep and AI.txt: Purpose-Based Scraping Controls

A robots.txt file answers only one question, whether a crawler may fetch this URL. That single bit no longer matches what site owners want to say in 2026. They want to allow search indexing but refuse model training, or to reserve text and data mining, and license it on request.