MAmmoTH-VL 2

Analysis

Why This Paper Matters

Large multimodal instruction datasets are critical for training vision-language models (VLMs) that can reason across text and images. However, manually curating such datasets is expensive and limited in scale. MAmmoTH-VL 2 tackles this bottleneck by introducing VisualWebInstruct, a fully automated pipeline that leverages web search engines and large language models (LLMs) to generate a diverse, high-quality dataset of nearly 900K question-answer pairs. This approach not only scales data collection but also ensures broad domain coverage—from math and physics to finance and biology—making it a practical solution for advancing VLM reasoning without human annotation.

How It Works

The VisualWebInstruct pipeline operates in two main stages: data mining from the internet and dataset refinement.

Stage 1: Mining Data from the Internet

The process begins with a carefully curated seed set of 30,000 images sourced from Stemez2, spanning disciplines like mathematics, physics, chemistry, and engineering. Using Google Lens, each seed image is used to search for visually similar content across the web, yielding approximately 60 URLs per image. After deduplication and filtering out non-educational domains, the pipeline retains 758,490 high-quality URLs.

For each URL, the HTML content is processed to build an accessibility tree—a clean, hierarchical representation that strips away navigation menus, ads, and other clutter, preserving only meaningful text and image elements. This structured format makes subsequent QA extraction efficient.

The Gemini 1.5 Flash model is then prompted to extract QA pairs from these accessibility trees. It identifies questions, relevant images, and detailed solutions, preserving mathematical notation and step-by-step reasoning. This yields 421,320 raw QA pairs, about 60% of which include images. A post-processing step using the same model filters out invalid or unclear pairs, retaining 361,015 valid QA pairs and 331,818 valid images.

Stage 2: Dataset Refinement

To further improve quality, the pipeline uses GPT-4o in a two-stage refinement. First, for each question and its associated images, GPT-4o generates four different answer variations. Then, it acts as a judge to check consistency among these variations, retaining only those where answers align. This produces 1.04 million consistency-verified QA pairs. Finally, Gemini-2.0-Flash compares GPT-4o's answers with the original web-extracted answers; if they disagree, the original is kept, otherwise the GPT-generated version is retained.

The resulting dataset spans multiple disciplines, with the largest categories being mathematics (35.6%), physics (19.5%), and finance (18.9%), as shown in the statistics.

Results

The authors fine-tune the existing MAmmoTH-VL checkpoint on VisualWebInstruct, creating MAmmoTH-VL2. The model uses a Qwen2.5-7B-Instruct language tower, a SigLip vision tower, and a projector, following the Llava-OneVision architecture. A data mixing strategy combines VisualWebInstruct with modified LLaVA-CoT data in a 9:1 ratio.

Evaluation on seven multimodal reasoning benchmarks (MMMU, MMMU-Pro, MathVista, MMVet, MathVerse, Dynamath) shows MAmmoTH-VL2 achieves an average accuracy of 50.4%, outperforming comparable open-source VLMs. Notably, it excels on MathVista (68.1%) and MathVerse (42.6%), and shows strong gains on MMMU-Pro-std (40.7%). It also competes with reasoning-enhanced models like LLaVA-CoT, achieving 26.3% on MMMU-Pro Vision versus LLaVA-CoTM's 23.7%.

However, MAmmoTH-VL2 still lags behind closed-source models like GPT-4o and Claude-3.5-Sonnet, indicating room for improvement.

Significance

VisualWebInstruct offers a scalable, automated alternative to manual dataset creation for multimodal instruction tuning. By leveraging web search and LLMs, it generates diverse, high-quality data across many domains, reducing human effort and cost. The resulting MAmmoTH-VL2 model demonstrates that this approach can significantly boost VLM reasoning capabilities, particularly in math and complex reasoning tasks. This work provides a practical blueprint for future dataset construction and opens the door for more capable open-source VLMs that can compete with proprietary systems.