MAmmoTH 2

Analysis

Why This Paper Matters

Large language models (LLMs) have become remarkably capable, but their reasoning abilities often depend on the quality and diversity of instruction-tuning data. Traditionally, creating such data requires expensive human annotation or synthetic generation from powerful models like GPT-4. MAmmoTH2 challenges this paradigm by showing that the open web itself is a vast, untapped reservoir of natural instruction-response pairs. By systematically harvesting and refining 10 million such pairs from Common Crawl, the authors demonstrate that LLMs can achieve substantial reasoning gains without any manual curation. This is significant because it suggests a scalable, low-cost path to improving LLM reasoning that can be continuously updated as the web evolves.

How It Works

The WebInstruct pipeline consists of three stages: recall, extraction, and refinement.

Recall from Common Crawl. The authors start by crawling 100K seed examples from educational websites and 100K negative samples from Common Crawl to train a fastText classifier. After three epochs of training with 256-dimensional vectors, the classifier recalls relevant documents from an internal Common Crawl snapshot, yielding 100 billion tokens. These documents are grouped by domain, and only domains with over 1,000 documents are kept (~600K domains). GPT-3.5 then filters these domains to about 50K that likely contain instruction data. A second fastText classifier is trained using samples from these selected domains as positives and general Common Crawl as negatives, recalling 40 billion tokens. GPT-4 further refines the domain selection, ultimately producing 18 million raw documents.

Q-A Pair Extraction. The raw HTML documents contain noise like ads and boilerplate. Rule-based filtering removes site information, ads, and HTML markup, significantly reducing document length. Then, Qwen-72B is prompted with in-context examples to identify natural Q-A pairs. About 30% of the recalled documents contain such pairs, resulting in roughly 5 million initial Q-A pairs. To avoid contamination with evaluation benchmarks, n-gram string matching (n=10) filters out any questions or answers that overlap with test sets.

Q-A Pair Refinement. The extracted pairs are refined using two LLMs: Mixtral-22B×8 and Qwen-72B. These models reformat the pairs and, if the answer lacks explanation, generate intermediate reasoning steps. Using two models increases dataset diversity. The final dataset contains 10 million Q-A pairs.

To further enhance diversity and quality, the authors also incorporate several open-source instruction datasets: OpenHermes 2.5, Code-Feedback, Math-Plus (an augmented version of MetaMathQA with 395K samples), and Orca-Math (200K). All samples are unified into a multi-turn instruction tuning format. Models including Mistral 7B, Mixtral 8×7B, Llama-3 8B, and Yi-34B are fine-tuned for two epochs with a maximum sequence length of 4096.

Results

The results are striking. MAmmoTH2-7B achieves a 14-point average performance boost over Mistral-7B across reasoning benchmarks. MAmmoTH2-8B improves Llama-3-8B-base by an average of 8.8 points. Gains are consistent across model sizes: Yi-34B’s MATH performance jumps by 19% after WebInstruct training. The MAmmoTH2-Plus models, which incorporate additional open-source data, achieve state-of-the-art results on TheoremQA, GPQA, and ARC-C for models under 10 billion parameters. MAmmoTH2-7B-Plus approaches the best-known results on MATH and GSM. Notably, MAmmoTH2-8B-Plus outperforms Llama3-Instruct (trained on a 10M instruction dataset) by an average of 6% across benchmarks. Scaling further, MAmmoTH2-8x7B-Plus matches the performance of Qwen-1.5-110B with only 13 billion active parameters.

Significance

MAmmoTH2 demonstrates that the web is a rich, largely untapped source of natural instruction data that can rival or surpass manually curated datasets. The pipeline is scalable, automated, and can be continuously updated as new web content emerges. This work has immediate practical implications: it reduces the dependency on expensive human annotation or proprietary model outputs for instruction tuning. The approach also highlights the importance of careful filtering and refinement to extract high-quality pairs from noisy web data. Future work could explore more sophisticated recall mechanisms, multilingual extraction, or integration with other data sources. For AI practitioners, MAmmoTH2 offers a blueprint for building powerful reasoning models with minimal manual effort, potentially democratizing access to state-of-the-art LLM capabilities.