OLMoE: An open, small, and state-of-the-art mixture-of-experts model

We’re introducing OLMoE, jointly developed with Contextual AI , which is the first mixture-of-experts model to join the OLMo family. OLMoE brings two important aspects to the space of truly open models — it is the first model to be on the Pareto frontier of performance and size, while also being released with open data, code, evaluations, logs, and intermediate training checkpoints. Over the last few years, mixture-of-experts architectures have become a core technology used by closed AI labs to more efficiently serve and train leading language models (LMs).