Qwen2-0.5B|1.5B|7B|57B-A14B-MoE|72B

Qwen2 is the evolution from Qwen1.5, a series of open-source language models developed by the Qwen Team (Alibaba Cloud). It includes pretrained and instruction-tuned models in five sizes: Qwen2-0.5B, Qwen2-1.5B, Qwen2-7B, Qwen2-57B-A14B (Mixture-of-Experts), and Qwen2-72B. All models now feature Group Query Attention (GQA) for faster inference and lower memory usage. Small models (0.5B and 1.5B) use tying embedding to reduce parameter overhead. The base models are pretrained on 32K token contexts with strong extrapolation, while the 7B-Instruct and 72B-Instruct models support up to 128K tokens when augmented with YARN. Training data spans 27 additional languages beyond English and Chinese, including Western and Eastern European, Middle Eastern, Asian, and South Asian languages, with explicit handling of code-switching. Qwen2 achieves state-of-the-art performance in benchmarks, with significant improvements in coding and mathematics over Qwen1.5. The models are released on Hugging Face and ModelScope under an open-source license.