Pass the Baton: Trajectory-Relayed On-Policy Distillation

Analysis

Why This Paper Matters

On-policy distillation (OPD) is a powerful technique for transferring knowledge from a large teacher model to a smaller student model by aligning the student's token-level predictions with the teacher's along the student's own generation trajectory. However, OPD suffers from a critical flaw: once the student makes a mistake early in generation, all subsequent tokens are built on that error, leading to misdirected supervision and wasted computation. This paper, "Pass the Baton: Trajectory-Relayed On-Policy Distillation," directly addresses this "prefix failure" problem with an elegant and practical solution.

The significance lies in the paper's identification of a simple, label-free signal—teacher-student continuation asymmetry on failed prefixes—that can be used to trigger a handoff. Instead of complex reward modeling or search, the method lets the teacher briefly take over generation at the point of failure, producing a correct continuation leg, and then hands back to the student. This is both computationally efficient and conceptually clean, making it highly relevant for practitioners training smaller models for reasoning tasks where early errors are common.

Technical Contributions

• Prefix Failure Detection: The paper identifies that when a student commits to a wrong reasoning direction, the teacher tends to redirect while the student continues along the original path. This asymmetry is used as a label-free trigger for intervention.
• Relay Trajectory Construction: At detected trigger points, the teacher generates a short "leg" (a few tokens), after which the student resumes generation. The entire relay trajectory is used for optimization, providing correct supervision at critical junctures.
• Limited Relay Budget: To avoid excessive deviation from the student policy, the method concentrates a limited relay budget on early positions where prefix failure is most damaging. This balances correction with policy preservation.
• Compute Efficiency: By reducing the length of misdirected trajectories, the method cuts training trajectory length by over 50%, directly translating to faster training.

Results

Using a Qwen3-4B-Instruct-2507 teacher and Qwen3-0.6B/1.7B-Non-Thinking students on eight mathematical reasoning benchmarks, Relay-OPD achieves the best or second-best results on every benchmark. For the 1.7B student, it outperforms standard OPD by +5.73% on average and the strongest baseline FastOPD by +1.49%. Gains are consistent for the 0.6B student as well. Notably, these improvements come with a >50% reduction in training trajectory length, demonstrating both accuracy and efficiency gains.

Significance

This work has immediate practical implications for AI practitioners working on distillation for reasoning tasks. By providing a simple, effective fix for a known weakness of OPD, Relay-OPD could become a standard component in distillation pipelines. The idea of using teacher-student asymmetry as a trigger is general and could extend beyond mathematical reasoning to other sequence generation tasks like code generation or dialogue. The compute savings are substantial, making it easier to train capable smaller models without sacrificing quality.