RM-R1

Analysis

Why This Paper Matters

Reward models are the backbone of reinforcement learning from human feedback (RLHF), but traditional approaches often act as black boxes—they output a scalar score without explaining why. This lack of interpretability makes it hard to trust or debug them. RM-R1 tackles this head-on by reframing reward modeling as a reasoning task. Instead of just scoring, the model generates a structured reasoning trace or a rubric, explaining its evaluation. This not only improves transparency but also boosts performance, as the model must "think" before judging. In an era where AI alignment is critical, RM-R1 offers a path toward more accountable and capable reward models.

How It Works

RM-R1 is a family of Reasoning Reward Models. The core idea is to train a generative model to produce a reasoning trace that justifies its evaluation of two candidate responses. The training pipeline has two stages: distillation and reinforcement learning.

Stage 1: Distillation of Reasoning Traces

Starting from any off-the-shelf instruction-tuned model (e.g., Qwen-2.5-14b-instruct), the authors synthesize high-quality reasoning traces. For each training sample (prompt, two responses, and a label indicating which is better), an "oracle" model like o3 or Claude-3-7-sonnet generates a structured reasoning trace that explains why the correct response is better. The model is then fine-tuned to maximize the likelihood of generating this trace and the correct judgment. This distillation unlocks basic reasoning ability for reward modeling.

Stage 2: Reinforcement Learning with Verifiable Rewards

Distilled models can overfit to patterns in the distillation data, limiting generalization. To address this, the authors apply reinforcement learning, specifically GRPO (Group Relative Policy Optimization), treating the reward model as a policy. They simplify the reward formulation from DeepSeek-R1 to focus only on correctness (format rewards didn't help). During rollout, a system prompt guides the model to classify the task as either "Reasoning" or "Chat". For reasoning tasks (math, coding, multi-step inference), the model first solves the problem itself, then evaluates responses against its own solution. For chat tasks, it generates a tailored rubric with weighted criteria and justifications. The final output is a structured evaluation with quotes and summaries, ending with a verdict in the format <answer>[[A]]</answer> or <answer>[[B]]</answer>.

The training data comes from three sources: Skywork Reward Preference 80K (filtered to remove spurious correlations from magpie_ultra), Code-Preference-Pairs (8K examples), and Math-DPO-10K. The magpie_ultra subset was excluded because it had a systematic bias where rejected responses always contained the token <im_start>, while accepted ones did not.

Results

RM-R1 achieves state-of-the-art performance on RewardBench and RM-Bench, outperforming larger commercial models like GPT-4 and Gemini. Notably, RM-R1-Qwen-Instruct-14B beats DeepSeek-GRM-27B, demonstrating the effectiveness of the reasoning-based training pipeline. The Qwen-2.5-Instruct based models generalize well across domains without obvious bias, while DeepSeek-Distilled-Qwen based models excel in reasoning-centric benchmarks (math and code). The approach is also data-efficient: Qwen-2.5-Instruct based RM-R1 achieves competitive results with only 8.7K training examples, compared to 800K for DeepSeek-GRM-27B.

Significance

RM-R1 represents a paradigm shift in reward modeling by making the evaluation process interpretable and reasoning-driven. This has immediate practical implications: developers can inspect why a model prefers one response over another, enabling better debugging and trust. The two-stage training pipeline (distillation + RL) is a scalable recipe that can be applied to any instruction-tuned model. By reducing reliance on human-labeled preferences and leveraging synthetic reasoning traces, RM-R1 could lower the cost of alignment. However, the approach depends on high-quality oracle traces and may still struggle with tasks requiring nuanced human judgment. Overall, RM-R1 opens the door to more transparent and capable reward models, which is crucial as AI systems become more autonomous.