Math Shepherd

Analysis

Why This Paper Matters

Large language models (LLMs) have shown impressive capabilities in mathematical reasoning, but verifying the correctness of their step-by-step solutions remains a challenge. Traditional outcome reward models (ORMs) only check the final answer, missing reasoning errors that happen to lead to the correct result. Process reward models (PRMs) offer finer-grained feedback by scoring each step, but they typically require expensive human annotations. Math Shepherd breaks this bottleneck by automatically generating step-level supervision data, making process reward models practical and scalable. This matters because it enables both better verification of LLM outputs and more effective reinforcement learning through step-by-step rewards, directly addressing a key limitation in current LLM training pipelines.

How It Works

Math Shepherd's core idea is to define the quality of a reasoning step by its potential to lead to the correct final answer. This is inspired by Monte Carlo Tree Search (MCTS): for each intermediate step in a candidate solution, the system generates multiple complete solutions starting from that step. By analyzing how many of these completions reach the correct answer, it assigns a score to the original step.

Task Formulation

Given a math problem, the system samples N candidate solutions from a generator. Each solution is scored by a reward model, and the highest-scoring solution is selected. The paper compares two types of reward models:

• Outcome Reward Model (ORM): Assigns a single score to the entire solution based on whether the final answer is correct. Training data can be automatically constructed by checking final answers, but this can mislabel solutions with correct answers but flawed reasoning.

• Process Reward Model (PRM): Assigns a score to each reasoning step, providing more granular feedback. Traditionally, PRMs require human-annotated step labels, which is costly.

Automatic Process Annotation

For each step s_i in a solution, the system uses a "completer" LLM to generate N complete solutions from that step onward. This yields a set of completions with their final answers. Two estimation methods are used to score the step:

• Hard Estimation (HE): A step is considered good if any completion reaches the correct answer. This is a binary score.

• Soft Estimation (SE): The step's score is the proportion of completions that reach the correct answer, providing a more nuanced quality measure.

These scores are used to train a PRM via cross-entropy loss, learning to predict step quality from the step itself.

Ranking for Verification

For verification, the overall score of a solution is taken as the minimum PRM score across all its steps (the weakest link). Solutions are grouped by their final answer, and the group with the highest combined score (from self-consistency and the reward model) is selected.

Reinforcement Learning with Process Supervision

Instead of providing a single reward at the end of a solution (as in standard PPO with ORM), Math Shepherd provides a reward after each reasoning step, derived from the trained PRM. This step-by-step PPO gives the model more granular feedback, helping it learn which steps are correct and which are not, leading to more efficient policy updates.

Results

• Math Shepherd consistently outperforms self-consistency and ORM as a verifier across multiple LLMs (LLaMA2, LLemma, Mistral, DeepSeek) on both GSM8K and MATH.
• The advantage of PRM over ORM is more pronounced on the harder MATH dataset, suggesting ORM's effectiveness is limited to simpler problems.
• Interestingly, combining self-consistency with a strong reward model can sometimes hurt performance, indicating careful integration is needed.

• Step-by-step PPO using Math Shepherd as a reward model significantly improves the performance of supervised fine-tuned LLMs, outperforming standard PPO with ORM.
• Reinforcement Fine-Tuning (RFT) shows limited gains, likely because the base models already benefited from data augmentation.

• Combining RL with verification yields complementary improvements. Using self-consistency as a verifier after RL with Math Shepherd gives better results than using the initial reward model alone, highlighting the need for stronger verification after RL.

Significance

Math Shepherd offers a practical, annotation-free path to process-level supervision for LLMs. By automating the creation of step-level training data, it removes a major bottleneck in developing process reward models. The dual use of these models—for both verification and reinforcement learning—demonstrates a unified framework that improves LLM reasoning at multiple stages. This approach is not limited to math; it can be extended to any domain where multi-step reasoning can be evaluated by the correctness of final outcomes. The paper's findings also caution against naive combinations of verification methods, providing guidance for future work. As LLMs are increasingly deployed in tasks requiring rigorous reasoning, Math Shepherd's methodology could become a standard component in training and inference pipelines.