Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
FreeImproving LLM reasoning by targeting high-entropy tokens in RLVR
FreeFree tier
About Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
This research paper, accepted to NeurIPS 2025, investigates Reinforcement Learning with Verifiable Rewards (RLVR) for LLM reasoning through a novel token entropy perspective. The authors observe that only a small fraction of tokens (high-entropy minority tokens) act as critical forks guiding reasoning directions, and RLVR primarily adjusts these tokens. By restricting policy gradient updates to these forking tokens, they achieve performance comparable to full updates on Qwen3-8B and significantly surpass them on larger models (e.g., +11.04 on AIME 2025 for Qwen3-32B), demonstrating a strong scaling trend and offering a more efficient approach to enhancing LLM reasoning.
Key Features
Token entropy analysis reveals high-entropy minority tokens as critical reasoning forks
RLVR training primarily adjusts high-entropy tokens while preserving base model patterns
Restricting policy gradient updates to forking tokens maintains performance with only 20% of tokens
Significant improvements on Qwen3 models (32B: +11.04 AIME'25, 14B: +4.79 AIME'25)
Demonstrates scaling trend: larger models benefit more from token-focused RLVR
Accepted to NeurIPS 2025 with 25 pages, 17 figures, 2 tables
Pros & Cons
Pros
- Novel token-entropy perspective provides clear mechanism understanding of RLVR
- Computationally efficient: only 20% of tokens need gradient updates for full performance
- Achieves state-of-the-art reasoning gains on AIME benchmarks for Qwen3 models
- Scales well with model size (larger gains on 32B than 8B)
- Open access paper with detailed figures and appendices
Cons
- Method validated only on Qwen3 model family; generalizability to other architectures unproven
- Requires access to token-level entropy calculations during training, increasing implementation complexity
- As a research paper, no ready-to-use code or library provided in the abstract
- Depends on verifiable rewards which may limit applicability to subjective tasks
Best For
Enhancing reasoning abilities of large language models via reinforcement learningEfficient fine-tuning of LLMs for mathematical and logical reasoning tasksUnderstanding token-level dynamics in chain-of-thought reasoning
FAQ
What is RLVR in this paper?
RLVR stands for Reinforcement Learning with Verifiable Rewards, a method used to enhance LLM reasoning by training on tasks where rewards are automatically computed (e.g., correctness of math answers).
How does token entropy affect reasoning?
The paper finds that high-entropy tokens act as forking points in chain-of-thought reasoning, steering the model toward different reasoning paths. RLVR primarily adjusts these tokens, and optimizing only them yields similar or better performance than full-gradient updates.
What are the main experimental results?
On Qwen3-32B, restricting updates to forking tokens outperformed full-gradient updates by +11.04 on AIME 2025 and +7.71 on AIME 2024. On Qwen3-14B, gains were +4.79 and +5.21 respectively, demonstrating a clear scaling trend.