Small datasets are just BERT with a math mask

Pre-trained models are built for the deluge. They thrive on the assumption that enough tokens will eventually smooth out the noise. But industrial datasets are rarely a deluge.