Reinforcement Learning via Self-Distillation

Analysis

Why This Paper Matters

This paper addresses a critical bottleneck in reinforcement learning for large language models (LLMs): the credit-assignment problem arising from scalar outcome rewards. In verifiable domains like code and math, environments often produce rich textual feedback (e.g., runtime errors, judge evaluations) that is discarded by current RLVR methods. By formalizing this as reinforcement learning with rich feedback (RLRF), the authors open a new direction for leveraging available information to improve learning efficiency.

The key insight is that the model itself can serve as a self-teacher when conditioned on feedback, eliminating the need for an external teacher or explicit reward model. This is particularly important for practitioners who want to post-tune LLMs without additional infrastructure. The method's ability to also work with only scalar feedback by using successful rollouts as implicit feedback further broadens its applicability.

Technical Contributions

• Formalization of RLRF: The paper clearly defines the setting where verifiable environments provide tokenized feedback beyond a scalar reward, distinguishing it from standard RLVR.
• Self-Distillation Policy Optimization (SDPO): The core innovation is treating the current policy conditioned on feedback as a self-teacher. The model generates next-token predictions after seeing feedback, and these predictions are distilled back into the policy without feedback. This leverages in-context learning to identify mistakes.
• Implicit feedback from successes: For environments with only scalar rewards, SDPO uses successful rollouts as implicit feedback for failed attempts, effectively creating a dense signal.
• Test-time acceleration: SDPO can be applied to individual questions at test time, reducing the number of attempts needed to discover correct solutions on binary-reward tasks.

Results

• On LiveCodeBench v6 (competitive programming), SDPO improves sample efficiency and final accuracy over strong RLVR baselines.
• Across scientific reasoning and tool use tasks, consistent gains are reported.
• For difficult binary-reward tasks, SDPO achieves the same discovery probability as best-of-k sampling or multi-turn conversations with 3x fewer attempts.
• The method also outperforms baselines in standard RLVR environments that only return scalar feedback, demonstrating robustness.

Significance

SDPO offers a practical, infrastructure-light approach to improve post-training of LLMs in verifiable domains. By converting rich feedback into a dense learning signal, it reduces the credit-assignment bottleneck and improves sample efficiency. This could accelerate development of LLMs for code generation, mathematical reasoning, and tool use. The self-distillation paradigm also suggests that models can learn from their own mistakes without external supervision, which may have broader implications for autonomous learning systems. However, the paper does not explore noisy or adversarial feedback scenarios, and the computational cost of the self-distillation loop is not quantified. Future work could investigate scaling to larger models and more complex feedback types.