RL for Reasoning in LLMs with One Training Example

Analysis

Why This Paper Matters

This paper presents a striking finding: reinforcement learning with verifiable reward (RLVR) can achieve dramatic improvements in LLM math reasoning using just a single training example. The result challenges the prevailing assumption that large-scale data is essential for RL-based fine-tuning. By matching the performance of a 1,200-example subset with one example, the authors open the door to extremely data-efficient reasoning enhancement, which could reduce computational costs and democratize access to advanced reasoning capabilities.

The phenomenon of post-saturation generalization—where test performance continues to improve even after training accuracy plateaus—is particularly intriguing. It suggests that the model is not simply memorizing but developing more robust reasoning strategies through continued exploration. This observation, along with the identification of cross-category generalization and increased self-reflection, provides new insights into how RL shapes internal reasoning processes.

Technical Contributions

• 1-shot RLVR framework: Demonstrates that a single verifiable-reward example can effectively incentivize reasoning, with careful tuning of entropy loss to promote exploration.
• Policy gradient vs. grokking: Controlled experiments confirm that the improvement stems from policy gradient loss, not the delayed generalization seen in grokking.
• Exploration via entropy: Shows that an appropriate entropy coefficient is critical; too little leads to premature convergence, too much prevents learning.
• Robustness across models and algorithms: Validates findings on Qwen2.5-Math-7B, Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B, and with both GRPO and PPO.
• Phenomenon cataloging: Documents cross-category generalization, increased self-reflection, and post-saturation generalization as emergent behaviors.

Results

• Qwen2.5-Math-1.5B: MATH500 accuracy jumps from 36.0% to 73.6% with 1-shot RLVR (8.6% beyond format correction). Average across six benchmarks rises from 17.6% to 35.7% (7.0% non-format gain).
• Comparison to 1.2k subset: 1-shot matches the 1.2k DeepScaleR subset (73.6% vs 73.6% on MATH500; 35.7% vs 35.9% average). Two examples slightly exceed (74.8%, 36.6%).
• Generalization: Improvements are not limited to the category of the training example; cross-category gains are observed.
• Post-saturation generalization: Test accuracy continues to rise even after training accuracy reaches 100%, indicating ongoing refinement.

Significance

This paper has the potential to reshape how practitioners approach RL-based reasoning fine-tuning. The extreme data efficiency suggests that the bottleneck may not be data quantity but the quality of the reward signal and the exploration strategy. It also raises important questions about the mechanisms of RLVR—why a single example can trigger such broad generalization. Future work could extend these findings to other domains (e.g., code generation, scientific reasoning) and explore whether even zero-shot RLVR is possible. The open-source release of resources will facilitate replication and further innovation.