Does RL Incentivize Reasoning Capacity in LLMs Beyond the Base Model

Analysis

Why This Paper Matters

This paper challenges a widely held assumption in the LLM community: that reinforcement learning from verifiable rewards (RLVR) inherently improves reasoning capabilities. By systematically comparing pass@k performance across a range of k values, the authors reveal a nuanced trade-off. At low k, RLVR indeed boosts sampling efficiency, making it easier to find correct answers with fewer attempts. However, at high k, the base model actually outperforms the RLVR-tuned model, suggesting that RLVR constrains the model's exploration of diverse reasoning paths. This finding is significant because it implies that current RLVR methods may inadvertently limit the model's reasoning capacity, potentially by overfitting to reward signals or narrowing the policy distribution. For practitioners, this means that RLVR should be applied with caution, especially when broad reasoning exploration is desired.

Technical Contributions

The paper's key innovation is the empirical demonstration of a capacity-efficiency trade-off in RLVR. Specifically:

• Pass@k analysis across k values: The authors evaluate both low and high k regimes, revealing opposite effects.
• Exploration reduction mechanism: They attribute the high-k degradation to reduced exploration of successful reasoning paths that exist in the base model.
• Challenging prevailing narrative: The work directly contradicts the common belief that RLVR universally improves reasoning.

Results

The paper reports that RLVR improves pass@k at low k (e.g., k=1, 5) but degrades it at high k (e.g., k=100, 1000). The exact metrics are not provided in the abstract, but the trend is clear: the base model's reasoning capacity boundary is higher when many samples are allowed. This suggests that RLVR sacrifices breadth for precision.

Significance

This work has broad implications for the AI field. It warns against over-reliance on RLVR as a panacea for reasoning improvement and encourages researchers to consider the exploration-exploitation trade-off in fine-tuning. It also opens avenues for developing RLVR methods that preserve or enhance exploration, potentially leading to more robust reasoning models. The paper's findings are particularly relevant for applications requiring diverse reasoning paths, such as open-ended problem solving or creative generation.