Investigating Pretraining Dynamics and Stability with OLMo Checkpoints

A central goal of the OLMo project is to use our experience to contribute to an open science of LM pretraining to provide a foundation for open-source pretraining efforts. For this reason, we are trying out something new: a short blog post where we use data from our open pretraining runs to evaluate how hypotheses about pretraining dynamics and stability manifest in our own OLMo checkpoints. We hope to both contribute to the open science of pretraining and better understand the root causes of instability in our own pretraining runs.