β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

Analysis

Why This Paper Matters

On-policy self-distillation (OPSD) has emerged as a promising technique for improving reasoning language models, but its practical adoption is hindered by brittleness and engineering complexity. This paper identifies a fundamental structural issue: vanilla OPSD is a special case of a broader policy-optimization family with a fixed KL penalty weight β=1. By generalizing this to a tunable β, the authors transform an implicit hyperparameter into a controllable regularization mechanism, offering a principled way to balance between staying close to a reference policy and leveraging privileged teacher guidance.

The significance lies in the elegant theoretical connection: the optimal policy for β-OPSD is a geometric interpolation between the reference and teacher policies. This insight allows the authors to bypass expensive RL optimization by directly using the closed-form solution as a distillation target. This not only reduces computational cost but also mitigates high variance, making the method more stable and practical. The paper thus provides a clear route from self-distillation to policy optimization and back, addressing a key bottleneck in RL fine-tuning of language models.

Technical Contributions

• Generalized Objective: Introduces β-OPSD, a family of objectives where β weights the KL penalty, unifying OPSD and policy optimization under one framework.
• Closed-Form Solution: Derives the optimal policy as a geometric interpolation between reference and teacher policies, enabling direct distillation targets.
• Logit Mixing: Implements β-OPSD by mixing token-level logits from reference and teacher, which is computationally efficient and approximates the RL solution.
• Return-to-Go Credit Assignment: Integrates return-to-go signals to align token-level updates with sequence-level objectives, improving optimization stability.
• Theoretical Equivalence: Shows that vanilla OPSD is exactly the β=1 case, providing a new perspective on existing methods.

Results

The paper reports experiments on mathematical reasoning benchmarks, where β-OPSD consistently outperforms vanilla OPSD. The improvements are observed in both optimization stability and downstream reasoning performance. While specific numerical metrics are not detailed in the abstract, the consistent gains across benchmarks suggest the method's robustness. The use of return-to-go credit assignment further enhances performance, indicating that the sequence-level alignment is beneficial.

Significance

β-OPSD offers a practical and theoretically grounded method for improving reasoning language models. By bridging self-distillation and policy optimization, it provides a cost-effective alternative to full RL fine-tuning, which is often resource-intensive. The ability to control the trade-off between reference and teacher guidance via β gives practitioners a new hyperparameter to tune for different tasks. This work could influence future research on efficient RL fine-tuning and self-improvement methods, potentially extending to other domains beyond mathematical reasoning. The theoretical insights may also inspire new algorithms that leverage closed-form solutions to approximate complex optimization objectives.