OpenCodeReasoning

Analysis

Why This Paper Matters

Large language models (LLMs) have shown remarkable ability in code generation, but their reasoning during coding—especially for complex competitive programming tasks—remains a bottleneck. OpenCodeReasoning tackles this by creating a massive, high-quality synthetic dataset that pairs code solutions with explicit reasoning traces. This is crucial because high-quality reasoning data is scarce and expensive to produce manually. By leveraging DeepSeek-R1 to generate these traces and then carefully filtering them, the authors provide a resource that can be used to fine-tune smaller models, making advanced code reasoning more accessible to the open-source community.

How It Works

The construction of OpenCodeReasoning proceeds in three main stages: question collection, solution generation, and post-processing refinement.

First, competitive programming questions are gathered from established sources: TACO, APPS, CodeContests, and CodeForces, all part of the OpenR1 project. Exact-match deduplication reduces the initial pool to 28,904 unique questions spanning various difficulty levels. To prevent data leakage with evaluation benchmarks, the authors compute cosine similarity (threshold 0.7) between each dataset question and benchmark questions, then use Llama-3.3-70B-Instruct and Qwen2.5-32B-Instruct as judges to assess semantic similarity. Manual inspection of 90 potentially problematic samples confirmed no paraphrases or semantic overlaps.

Next, multiple solutions per question are generated using DeepSeek-R1, primarily in Python, with some C++ experiments for the harder IOI benchmark. Sampling uses nucleus sampling (temperature 0.6, top-p 0.95) and explicitly injects a <think> tag to force the model to produce reasoning traces.

The post-processing pipeline is critical for quality. First, solutions are verified to contain reasoning traces enclosed in <think> and </think> tags. Then, the reasoning traces are extracted and checked for code blocks (delimited by python ... or cpp ... ). Responses where reasoning traces themselves contain code blocks are filtered out. Finally, the syntactic correctness of the solution code blocks is verified by parsing. This refinement yields 736,712 Python samples and 355,792 C++ samples.

The dataset is scaled incrementally from 25k to 736k samples. Initial scaling from 25k to 100k samples using CodeContests questions showed substantial gains on LiveCodeBench. Noticing an imbalance favoring easy and medium problems, the authors then focused on the hard subset of CodeContests (about 4.5k instructions) and generated multiple solutions. Finally, additional instructions were integrated to compose the full 28k unique question set, with solutions generated for the new questions.

Results

The authors fine-tune Qwen2.5 base and instruct models at 7B, 14B, and 32B parameters for 3 epochs with a maximum sequence length of 32,768. Evaluation on LiveCodeBench and CodeContests shows:

• Competitive at Small Scales: OCR-Qwen-7B and OCR-Qwen-7B-Instruct outperform baseline models in the 7B category, with the instruct version exceeding the best baseline by a significant margin.
• Scaling Improves Performance: Performance improves significantly from 7B to 14B to 32B for all models, especially OCR-Qwen models.
• Near State-of-the-Art Performance: At 32B, OCR-Qwen models outperform strong competitors and achieve near state-of-the-art results, only slightly behind DeepSeek-R1.

Significance

OpenCodeReasoning demonstrates that high-quality synthetic reasoning data can be generated at scale and effectively distilled into smaller models. This approach reduces reliance on proprietary models for code reasoning and provides a blueprint for creating similar datasets in other domains. The careful deduplication and post-processing ensure data quality, while the scaling experiments show clear performance gains. The near state-of-the-art results at 32B parameters suggest that with sufficient data, smaller models can approach the capabilities of much larger ones, democratizing access to advanced code reasoning.