DeepMath

Analysis

Why This Paper Matters

Mathematical reasoning remains a critical frontier for large language models (LLMs), yet progress is often bottlenecked by the quality and diversity of training data. Existing datasets like GSM8K and MATH are either too easy or contaminated with benchmark overlaps, leading to inflated evaluation scores and poor generalization. DeepMath-103K addresses this gap by offering a large-scale, challenging, and rigorously decontaminated dataset specifically designed for training reasoning models via reinforcement learning (RL) or supervised fine-tuning (SFT). Its focus on verifiable answers and multiple solution paths makes it uniquely suited for rule-based RL and reward modeling, pushing the boundaries of what LLMs can achieve in complex math.

How It Works

The construction of DeepMath-103K follows a meticulous four-stage pipeline to ensure quality and integrity.

First, the authors analyzed existing open-source math datasets (e.g., MMIQC, WebInstructSub, NuminaMath-CoT) and identified that many were skewed toward low difficulty. They collected 2.9 million raw questions from these sources after basic filtering.

Each sample in the final dataset includes: a question, a verifiable final answer, a difficulty score (3–10), hierarchical topic classification (e.g., Prealgebra, Calculus), and three distinct reasoning paths generated by DeepSeek-R1. The difficulty distribution is heavily weighted toward levels 5–10, with 95K problems in that range, making it significantly harder than prior datasets.

A key innovation is the rigorous decontamination process. The authors used a two-step semantic approach: first, embedding similarity search with paraphrase-multilingual-MiniLM-L12-v2 to find top-5 similar examples from 14 benchmark test sets; second, an LLM-Judge (Llama-3.3-70B-Instruct) compared each candidate with its top-5 matches twice (with swapped order) to detect paraphrases or duplicates. This caught subtle overlaps that simple n-gram methods would miss, as shown in the contamination rates of source datasets (up to 90%).

After decontamination, difficulty filtering retained only problems with level 5 or higher, assigned by GPT-4o using AoPS guidelines (averaged over six queries). Finally, answer verification ensured that all three R1-generated solutions produced identical answers, filtering out ambiguous or open-ended problems.

Results

Experiments show that DeepMath-103K significantly boosts mathematical reasoning. Starting from Qwen-2.5-7B-Base and R1-Distill-Qwen-1.5B, the authors applied SFT (using one or three solutions) and RL-Zero with binary rewards. DeepMath-Zero (RL-Zero on DeepMath-103K) achieved state-of-the-art performance on AIME, AMC, and MATH500, outperforming models trained on other RL datasets like Open-R1 and DAPO-17K.

Qualitative analysis revealed that DeepMath-Zero-7B generates longer, more detailed reasoning steps and exhibits increased use of beneficial cognitive behaviors like subgoal setting, verification, backtracking, and enumeration. This suggests the model learns more structured problem-solving strategies rather than relying on pattern matching.

Significance

DeepMath-103K sets a new standard for math reasoning datasets by combining scale, difficulty, decontamination, and verifiability. Its design enables diverse training paradigms—SFT, distillation, RL-Zero, and reward modeling—making it a versatile resource for the community. The rigorous decontamination ensures that benchmark results reflect genuine reasoning ability rather than data leakage. This dataset is likely to accelerate progress in LLM-based mathematical reasoning, especially for smaller models, and could inspire similar approaches in other domains requiring verifiable reasoning.