OLMoE: Open Mixture-of-Experts Language Models
FreeOpen Mixture-of-Experts Language Model with 1B active parameters, outperforming larger models.
FreeFree tier
Inputs: textOutputs: text
About OLMoE: Open Mixture-of-Experts Language Models
OLMoE is a fully open, state-of-the-art language model that uses a sparse Mixture-of-Experts (MoE) architecture. The OLMoE-1B-7B model has 7 billion total parameters but activates only 1 billion per input token, achieving high efficiency. It was pretrained on 5 trillion tokens and further fine-tuned to create the OLMoE-1B-7B-Instruct variant. The model outperforms all available models with similar active parameters, including larger models like Llama2-13B-Chat and DeepSeekMoE-16B. The project is fully open-source, releasing model weights, training data, code, and training logs.
Key Features
Sparse Mixture-of-Experts architecture with 7B total parameters and 1B active per token
Pretrained on 5 trillion tokens
Fully open-source: model weights, training data, code, and logs are publicly available
OLMoE-1B-7B-Instruct variant for instruction following
Outperforms Llama2-13B-Chat and DeepSeekMoE-16B in evaluations
Detailed analysis of expert routing demonstrating high specialization
Pros & Cons
Pros
- Fully open-source with all artifacts released
- High performance relative to active parameter count
- Outperforms larger dense and MoE models
- Transparent training and routing analysis
Best For
Language modeling and text generationInstruction-following chatbots and conversational AIResearch on Mixture-of-Experts training and routingEfficient inference with low active parameter count