Diffusion Language Models, From Scratch to Production

“Language model” used to be a very general term for statistical models trained on language tasks. It included everything from n-gram models to LSTMs and GRUs. But following the release of GPT-3.5 and ChatGPT, the term “language model” narrowed, at least in the popular sense, to mean one thing: autoregressive Transformers.