H2OVL-Mississippi

Analysis

Why This Paper Matters

In the race toward ever-larger vision-language models, the H2OVL-Mississippi collection offers a refreshing counterpoint: small, efficient models that punch above their weight on text-centric tasks. With parameter counts of just 0.8 billion and 2 billion, these models are designed for real-world deployment where computational resources are limited. Their standout performance on OCR and document understanding benchmarks shows that specialization, not just scale, can drive meaningful progress.

How It Works

The H2OVL-Mississippi models follow a ViT-MLP-LLM architecture inspired by the LLaVA series. The vision encoder is InternViT-300M, paired with either Danube-2 (1.8B) or Danube-3 (500M) as the language model. An MLP projector bridges the two components.

A key innovation is the dynamic resolution strategy. Images are divided into 448x448 tiles, with the number of tiles ranging from 1 to 6 depending on the image's aspect ratio and resolution. This produces between 256 and 1,590 visual tokens. A pixel shuffle operation reduces tokens per tile to 256, cutting computational cost without losing information.

The 2B model also employs Multi-Scale Adaptive Cropping (MSAC) to address the "sawtooth effect" from traditional cropping. MSAC generates multi-scale representations, capturing features at different scales—especially useful for small or irregularly shaped objects in documents. A resized 448x448 version of the full image is also provided to preserve overall layout context.

Training follows a two-stage approach: pre-training aligns visual and textual features, then fine-tuning focuses on task-specific modeling. For the 0.8B model, pre-training uses 11 million conversation examples across general QA, captioning, OCR, and reasoning. The MLP is trained first (with ViT and LLM frozen), then both MLP and LLM are jointly optimized. Fine-tuning uses 8 million examples exclusively for OCR tasks like text recognition and document parsing.

The 2B model's pre-training uses 5 million conversation pairs covering OCR, image captioning, and text-only data. Only the vision encoder and MLP are trained during this phase. Fine-tuning uses 12 million examples spanning general QA, OCR, document understanding, reasoning, captioning, and chart/table understanding.

Results

On OCRBench, the 0.8B model achieves the highest text recognition score, outperforming much larger models like InternVL2-26B and MiniCPM-V2.6. The 2B model also excels across text recognition, scene text VQA, document-oriented VQA, and key information extraction (KIE).

On text-oriented VQA benchmarks (TextVQA, DocVQA, InfoVQA), the 2B model achieves scores comparable to or better than larger models like Cambrian-13B. For document understanding, it achieves the second-highest accuracy on receipt processing (82), outperforming InternVL2-40B and GPT-4o. It also shows competitive results on driver's licenses and checks.

On general vision-language benchmarks, the 2B model performs competitively, excelling on Math Vista and OCRBench, though slightly behind Qwen2-VL-2B on MMBench and MMStar.

Significance

H2OVL-Mississippi shows that small models can achieve remarkable performance on specialized tasks when designed with the right architecture and training data. This is especially important for OCR and document analysis, where accuracy on text extraction and layout understanding is critical. The models are open-source and available on HuggingFace, enabling widespread adoption and further research. This work encourages a shift toward efficient, task-specific models that can run on edge devices, democratizing access to advanced AI capabilities.