AI's 'Creepy' Crawlers Criticized by Linux Foundation's IT Infrastructure Director

The Linux Foundation’s director of IT infrastructure says they now spend more CPU cycles “rendering commits for scrapers than we spend on all other kinds of legitimate access.” At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html…. [W]hen a source is guaranteed to be LLM-free, like the entire history of kernel commits, it’s worth its weight in gold as a source of training data… At the time of writing, linux.git is about 1.48 million commits.