Preprint
Large Language Models

DAPO: An Open-Source LLM Reinforcement Learning System at Scale

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Honglin Yu, Weinan Dai, Yuxuan Song, Xiang Wei, Haodong Zhou, Jingjing Liu, Wei Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yong-Xu Wu, Mingxuan Wang
March 18, 2025Neural Information Processing Systems2,289 citations
arXivPDFDOI

2.3k

Citations

492

Influential Citations

Neural Information Processing Systems

Venue

2025

Year

Abstract

Inference scaling empowers LLMs with unprecedented reasoning ability, with reinforcement learning as the core technique to elicit complex reasoning. However, key technical details of state-of-the-art reasoning LLMs are concealed (such as in OpenAI o1 blog and DeepSeek R1 technical report), thus the community still struggles to reproduce their RL training results. We propose the $\textbf{D}$ecoupled Clip and $\textbf{D}$ynamic s$\textbf{A}$mpling $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{DAPO}$) algorithm, and fully open-source a state-of-the-art large-scale RL system that achieves 50 points on AIME 2024 using Qwen2.5-32B base model. Unlike previous works that withhold training details, we introduce four key techniques of our algorithm that make large-scale LLM RL a success. In addition, we open-source our training code, which is built on the verl framework, along with a carefully curated and processed dataset. These components of our open-source system enhance reproducibility and support future research in large-scale LLM RL.

Analysis

Why This Paper Matters

DAPO addresses a critical gap in the field of large language model (LLM) reasoning: the lack of reproducible reinforcement learning (RL) training details. While models like OpenAI's o1 and DeepSeek R1 have demonstrated impressive reasoning abilities, their technical reports omit key implementation specifics, hindering the community's ability to replicate and advance these results. DAPO breaks this trend by fully open-sourcing its algorithm, training code, and dataset, making large-scale RL for LLMs accessible to researchers worldwide.

The paper's significance is underscored by its rapid citation count (2289), reflecting the community's urgent need for transparent RL methodologies. By achieving 50 points on AIME 2024 with a 32B parameter model, DAPO proves that open-source systems can rival closed-source performance, democratizing cutting-edge AI research.

Technical Contributions

  • Decoupled Clip: Separates the clipping mechanisms for policy and value losses, preventing instability during training and improving sample efficiency.
  • Dynamic Sampling: Adjusts the sampling distribution of prompts based on group reward statistics, focusing training on examples that yield high variance or difficulty.
  • Token-Level Loss: Applies policy gradient loss at the token level rather than sequence level, providing finer-grained credit assignment and improving learning signal.
  • Overlong Reward Shaping: Modifies reward shaping to avoid penalizing overly long generations, which is crucial for reasoning tasks where longer chains of thought are often beneficial.
  • Open-Source System: Built on the verl framework, the system includes complete training code and a curated dataset, enabling end-to-end reproducibility.

Results

The paper reports that DAPO achieves 50 points on the AIME 2024 benchmark using the Qwen2.5-32B base model. This is a notable result, as AIME is a challenging competition-level math dataset. The open-source nature of the system allows independent verification and further optimization. The high citation count suggests that the results have been widely adopted and validated by the community.

Significance

DAPO's open-source approach sets a new standard for transparency in LLM RL research. By releasing all components, it enables researchers to not only reproduce results but also to experiment with variations, accelerating innovation. This could lead to more robust reasoning models and a deeper understanding of RL dynamics in large-scale settings. Moreover, the techniques introduced—such as decoupled clipping and dynamic sampling—may generalize to other RL applications beyond LLMs, influencing the broader AI field. As closed models continue to dominate, DAPO serves as a crucial counterbalance, ensuring that the benefits of advanced reasoning capabilities are not confined to a few organizations.

reinforcement-learninglarge-language-modelsreasoningopen-sourcepolicy-optimizationscaling
Reinforcement LearningReasoningScaling Laws

Related Papers