AceReason-Nemotron 1.1

Analysis

Why This Paper Matters

Large language models have made impressive strides in reasoning, but achieving high performance on math and code benchmarks typically requires massive models or extensive compute. AceReason-Nemotron 1.1 shows that with careful orchestration of supervised fine-tuning (SFT) and reinforcement learning (RL), even a 7B parameter model can reach state-of-the-art results on challenging benchmarks like AIME25 and LiveCodeBench v6. The key insight is that scaling the SFT data—especially the number of unique prompts—combined with a stage-wise RL training pipeline that progressively increases response length budgets, can unlock substantial reasoning gains. This is particularly important for practitioners who need efficient, deployable models without sacrificing accuracy.

How It Works

The pipeline consists of two main phases: SFT and RL. For SFT, the authors collect a large dataset of 383K prompts (247K math, 136K code) from diverse sources, deduplicate, and decontaminate by removing samples with 9-gram overlaps with test sets. Responses are generated using DeepSeek-R1. To balance difficulty, they filter out prompts with responses around or below 2,000 tokens, keeping a mix of harder problems. The SFT phase itself is studied in detail: seven datasets (v1 to v7) are created, scaling from 36K to 2.2M samples. A multiple linear regression analysis reveals that increasing the number of unique prompts (coefficient a=4.831) has a larger impact on accuracy than increasing the number of responses per prompt (coefficient b=2.635), with an R² of 0.989.

For RL, the authors use the GRPO algorithm with on-policy training, generating 8 or 16 rollouts per question in a global batch of 128 prompts. They remove the KL divergence term and use a token-level policy gradient loss that rewards longer correct answers and penalizes longer incorrect ones. The training is stage-wise, starting with math-only stages at increasing response length budgets (8K, 16K, 24K) and then code-only stages (24K, 32K), followed by a final math-only stage at 32K. The temperature parameter is carefully tuned: a moderate temperature of 0.85, which yields an entropy around 0.26 initially and rises to 0.38 during training, balances exploration and exploitation. Inference uses a lower temperature of 0.6.

Results

The SFT model alone already outperforms several baselines, including Llama-Nemotron-Nano-8B-v1 and DeepSeek-R1-Distill-Qwen-7B. Applying the RL recipe yields substantial gains: on AIME24, AIME25, LiveCodeBench v5, and LiveCodeBench v6, the RL model achieves the highest accuracy among 7B-scale models. Notably, RL improves pass@K scores even at large K values (up to 128), with coding benchmarks showing a 5% improvement at K=128 on LiveCodeBench v6. The model also solves over ten additional hard coding problems that the SFT baseline could not solve within 128 attempts.

Significance

AceReason-Nemotron 1.1 provides a clear, reproducible recipe for combining SFT and RL to boost reasoning in small models. The findings on data scaling—especially the primacy of unique prompts over multiple responses per prompt—offer practical guidance for dataset curation. The stage-wise RL approach with careful temperature tuning demonstrates that even strong SFT models can be significantly improved through RL, and that the gains are concentrated on hard problems. This work is a valuable resource for AI practitioners aiming to build efficient reasoning systems, and the open-source release of models and data on HuggingFace facilitates further research and application.