SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

Analysis

Why This Paper Matters

Reinforcement learning with verifiable rewards (RLVR) has become a standard approach for fine-tuning language models on tasks with objective correctness signals, such as math and code. However, RLVR broadcasts a single response-level reward to all tokens, providing sparse and coarse feedback. On-policy distillation (OPD) offers a complementary dense advantage by scoring each token against a stronger teacher, but it caps performance at teacher quality and discourages exploration. Combining these two signals is intuitively appealing, yet naive fusion with a fixed coefficient leads to entropy collapse, as the authors demonstrate. This paper addresses a critical practical issue: how to effectively fuse dense and sparse advantage signals without destabilizing training. The proposed SAF framework is lightweight and modular, making it easy to integrate into existing RLVR pipelines. This is particularly relevant as the field moves toward more efficient and stable training of reasoning models.

Technical Contributions

• Identification of two miscalibrations: The paper pinpoints magnitude mismatch (OPD advantages can spike far beyond RLVR's bounded advantage, erasing its signal) and temporal mismatch (sustained full-strength OPD pulls the student toward the teacher, limiting exploration). These insights are crucial for understanding why naive fusion fails.
• Four-stage pipeline: SAF applies a sparsify-then-compress mechanism for magnitude control and a warm-up-then-anneal mechanism for temporal control, each independently switchable. This modular design allows practitioners to adapt the framework to their specific needs.
• Negligible overhead: The pipeline is lightweight, adding minimal computational cost, which is essential for large-scale training.
• Compatibility with GRPO: SAF is instantiated with GRPO, a popular RLVR algorithm, demonstrating its practical applicability.

Results

The authors evaluate SAF on seven mathematical reasoning and code generation benchmarks using Qwen3-1.7B/4B/8B models. Across all six model-domain settings, SAF consistently outperforms fixed-coefficient GRPO+OPD fusion, with aggregate score improvements ranging from 0.51% to 2.70%. Notably, SAF avoids entropy collapse, which is a common failure mode in RLVR+OPD fusion, and achieves more stable training. These results are robust across model sizes and domains, indicating the generalizability of the approach.

Significance

SAF offers a practical solution to a pressing problem in RLVR: how to leverage dense teacher signals without sacrificing exploration or stability. By providing a simple, modular framework, it enables practitioners to improve performance on reasoning and code generation tasks with minimal overhead. The insights into magnitude and temporal mismatches are broadly applicable to other advantage fusion scenarios, potentially influencing future research on multi-signal reinforcement learning. As RLVR becomes more prevalent in training large language models, methods like SAF will be essential for achieving stable and effective learning.