Optimizing Large-Scale Pretraining at Character.ai

Before Character.ai shifted its focus toward building on open-source model foundations, the company’s early pretraining team explored a range of techniques to make large-scale transformer training faster and more efficient. That work - led in part by our cofounder Noam Shazeer - is now being shared publicly