Reinforced Self-Training (ReST)

Analysis

Why This Paper Matters

Aligning large language models (LLMs) with human preferences is a central challenge in AI deployment. Traditional RLHF methods often rely on online reinforcement learning, which is computationally expensive, unstable, and prone to reward hacking. Reinforced Self-Training (ReST) offers a refreshingly simple alternative: it decouples data collection from policy improvement into separate offline stages, making alignment more stable, scalable, and easier to debug. By iteratively generating samples from the current policy and filtering them with a learned reward model, ReST achieves strong results on machine translation tasks while avoiding many pitfalls of online RL. This matters because it provides a practical, low-overhead pathway for practitioners to improve model quality without the complexity of full RLHF pipelines.

How It Works

ReST operates in two alternating loops: an outer Grow step and an inner Improve step. The process starts with an initial supervised policy (trained via negative log-likelihood on a dataset of input-output pairs).

In the Grow step, the current policy generates multiple output sequences for each input context, creating an augmented dataset. Each sample is scored by a learned reward model (trained on human preferences). This step corresponds to the acting phase in RL, but it is performed entirely offline—no environment interaction is needed.

In the Improve step, the augmented dataset is filtered to include only samples with rewards above a threshold τ. The policy is then fine-tuned on this filtered subset using an offline RL objective, such as behavior cloning (BC) loss or other offline RL losses. Crucially, the filtering threshold is increased across successive Improve steps (τ₁ < τ₂ < ... < τₙ), so each iteration uses a smaller but higher-quality subset. To prevent overfitting on small datasets, fine-tuning starts from the previous policy with a lower learning rate. This iterative refinement ensures policy improvement without generating new data each time—amortizing the cost of the Grow step over multiple Improve steps.

The algorithm is simple, stable, and has few hyperparameters. It also allows easy inspection of data quality and diagnosis of issues like reward hacking, since the Grow and Improve steps are decoupled.

Results

ReST was evaluated on three machine translation datasets: IWSLT 2014, WMT 2020, and a Web Domain dataset. Key findings include:

• Multiple Improve steps consistently increase reward model scores across all datasets, demonstrating the benefit of iterative filtering.

• Additional Grow steps further improve performance, showing that expanding the training data with better policies yields compounding gains.

• ReST significantly outperforms supervised learning baselines, even after just one Grow step, across different language pairs.

• Among offline RL losses, BC loss generally performs best within the ReST framework.

• ReST benefits from best-of-N sampling at inference time, similar to supervised models, indicating it maintains sample diversity.

• Compared to online RL (PPO), ReST achieves higher rewards with similar data and avoids the significant BLEU score drop observed with PPO, suggesting less reward hacking.

• Human evaluation confirms that all ReST variants outperform the baseline BC model. However, the ranking based on human judgment differs from that based on the learned reward model, highlighting that the reward model is an imperfect proxy—especially as the policy diverges from the behavior model.

Significance

ReST provides a practical, stable, and computationally efficient method for aligning LLMs with human preferences. By decoupling data generation from policy improvement, it reduces the complexity and instability of online RLHF while still enabling iterative improvement. The approach is particularly valuable for practitioners who need a simple, reproducible pipeline for fine-tuning models on human feedback. Its success on machine translation suggests broad applicability to other sequence generation tasks, such as summarization, dialogue, and code generation. The main caveat is the reliance on a learned reward model, which may not perfectly capture human preferences—especially as the policy explores new behaviors. Future work could explore combining ReST with more robust reward modeling or human-in-the-loop validation to further bridge the gap between automated metrics and true human satisfaction.