Training on synthetic web text can reduce language-model quality under standard compute budgets, researchers report

Mohit Iyyer says experiments across hundreds of pretrained models found that adding AI-generated web text usually underperformed adding human-written text.