MAmmoTH-VL

Analysis

Why This Paper Matters

Multimodal large language models (MLLMs) have made impressive strides, but their reasoning capabilities are often bottlenecked by the quality of instruction-tuning data. Existing datasets tend to focus on simplistic tasks with phrase-level answers, lacking the intermediate rationales that foster chain-of-thought (CoT) reasoning. MAmmoTH-VL tackles this head-on by introducing a scalable, cost-effective pipeline that generates a massive 12M instruction-response dataset rich in detailed reasoning. What makes this work particularly impactful is its commitment to open-source principles: the entire pipeline uses only open models and publicly available data, making it accessible to researchers and practitioners without massive budgets. The resulting model, MAmmoTH-VL-8B, achieves state-of-the-art performance among open-source MLLMs, especially in mathematical reasoning, demonstrating that high-quality data curation can rival the benefits of scaling model size.

How It Works

The MAmmoTH-VL pipeline consists of three key stages: data collection and categorization, task-specific rewriting, and self-filtering. First, the authors source raw image-text pairs from 153 publicly available multimodal datasets, covering a wide range of domains like OCR, charts, captions, and medical images. To manage this diversity, they reorganize the data into 10 major categories (e.g., General, OCR, Chart, Code&Math) and three quality groups (A, B, C) based on the informativeness and structure of existing responses. Group A datasets with high-quality responses are kept as-is, while Group B datasets with brief but improvable responses are targeted for rewriting. Group C datasets, deemed too vague, are discarded.

The rewriting step is where the magic happens. For caption-based data, the authors use a text-only model (Llama-3-70B-Instruct) to generate task-oriented Q&A pairs, leveraging its strength in creating diverse and complex instructions from rich textual descriptions. For all other data types, they employ a multimodal model (InternVL2-Llama3-76B) to ensure strong alignment between visual content and generated instructions. Customized prompts are designed for each data category to elicit the desired reasoning format.

A critical innovation is the self-data filtering step. Manual inspection of rewritten data revealed hallucinations, especially in OCR and chart tasks. To address this, the authors use the same multimodal model (InternVL2-Llama3-76B) as a judge to evaluate the logical consistency of each Q&A pair against the corresponding image. The insight is that while the model may generate inaccuracies, it is more reliable at verification. This "Model-as-Judge" approach efficiently filters out hallucinated or irrelevant content, ensuring high data quality.

The model architecture follows LLaVA-OneVision, with Qwen2.5-7B-Instruct as the language backbone, Siglip-so400m-patch14-384 as the vision encoder, and a two-layer MLP projector. Training proceeds in three stages: language-image alignment (using the LLaVA pre-training corpus), visual instruction tuning on 10M single-image samples from MAmmoTH-VL-12M, and finally one-vision tuning on a mixture of single-image, multi-image, and video data (2M samples).

Results

MAmmoTH-VL-8B achieves state-of-the-art performance among open-source MLLMs across a wide range of benchmarks. It particularly shines in mathematical reasoning, outperforming both open-source and some open-weight models on 9 benchmarks. The single-image variant (SI) shows the strongest results, while the multi-image/video version maintains robust performance with only slight decreases on certain tasks.

For multi-image and video tasks, MAmmoTH-VL-8B significantly outperforms prior open-source models like LLaVA-One-Vision-7B, achieving a 13-point improvement on MuirBench. However, it still lags behind Qwen2-VL-7B, which the authors attribute to the limited size of the multi-image/video training dataset (only 1M samples) due to computational constraints. This suggests that further scaling of multi-image/video data could yield additional gains.

Significance

MAmmoTH-VL demonstrates that high-quality, reasoning-rich instruction data can be generated at scale using only open models, democratizing access to advanced MLLM training. The pipeline's cost-effectiveness and reliance on open-source tools make it a valuable blueprint for the research community. By showing that careful data curation can unlock strong reasoning capabilities in a relatively small 8B model, the paper challenges the notion that massive model sizes are necessary for state-of-the-art performance. The work also highlights the importance of self-filtering to mitigate hallucinations, a persistent challenge in synthetic data generation. As the field moves toward more capable and trustworthy MLLMs, MAmmoTH-VL provides both a practical dataset and a methodological framework that can be adapted and extended for future research.