NVLM

Analysis

Why This Paper Matters

Multimodal large language models (LLMs) have become the backbone of AI applications that need to understand both text and images. However, two major challenges persist: how to efficiently process high-resolution images without losing detail, and how to prevent multimodal training from degrading the model's original text-only abilities. NVLM tackles both head-on. It introduces a family of models that not only rival proprietary systems like GPT-4o but also improve on text-only benchmarks after multimodal training—a rare feat. The paper's systematic comparison of decoder-only, cross-attention, and hybrid architectures provides a clear design space for practitioners. The 1-D tile-tagging method is a simple yet powerful innovation that boosts OCR and reasoning by giving the LLM explicit positional cues for image tiles. This work is a significant step toward open, frontier-class multimodal AI.

How It Works

NVLM comes in three flavors, each using the same frozen vision encoder (InternViT-6B-448px) and dynamic high-resolution (DHR) tiling. The DHR approach divides an input image into up to 6 tiles of 448×448 pixels, plus a thumbnail for global context. Each tile generates 1024 tokens, which are downsampled to 256 via pixel shuffle (concatenating four neighboring tokens along channels).

NVLM-D (Decoder-only): A 2-layer MLP projects image tokens to the LLM's hidden space. During SFT, both the MLP and LLM are trained while the vision encoder stays frozen. The key innovation is the 1-D tile tag: a text token like <tile_1> inserted before each tile's 256 tokens. This tells the LLM which tile is being processed, avoiding confusion from concatenating all tokens without delimiters. Ablations show 1-D tags outperform no tags, 2-D grid tags, and bounding-box tags, especially on OCR tasks.

NVLM-X (Cross-attention): Instead of feeding all image tokens into the LLM decoder, NVLM-X uses gated cross-attention layers inserted every 8 self-attention layers. This avoids the perceiver resampler, which can disrupt spatial relationships needed for OCR. Tile tags are also used, with each tag attending only to its corresponding image tokens via a custom attention mask. This architecture is more parameter-efficient for high-resolution images because it doesn't unroll all tokens in the decoder.

NVLM-H (Hybrid): The best of both worlds. Thumbnail tokens are processed via self-attention (like NVLM-D) for joint multimodal reasoning, while regular tiles are processed via cross-attention (like NVLM-X) for efficiency. This hybrid path enhances high-resolution capability without the computational cost of full decoder unrolling.

Training is two-stage: pretraining (only modality-alignment modules trained, vision and LLM frozen) and SFT (LLM and alignment modules trained, vision frozen). A high-quality text-only SFT dataset (ShareGPT, SlimOrca, math, code) is blended to maintain text performance.

Results

NVLM-D 72B achieves top scores on OCRBench (853), VQAv2 (85.4), and MMMU (59.7), outperforming proprietary models like GPT-4o and open-access models like InternVL-2. NVLM-H 72B reaches the highest MMMU (60.2) and MathVista (66.6) among open-access models. Critically, all NVLM models show improved text-only performance over their backbone Qwen2-72B-Instruct, unlike other multimodal LLMs that degrade. This is attributed to the high-quality text SFT data and substantial multimodal math training.

Significance

NVLM is a landmark for open-access multimodal AI. It proves that with careful architecture design (hybrid, tile tags) and data curation (text-only SFT blending), open models can match or exceed proprietary ones. The hybrid architecture offers a practical trade-off: NVLM-X trains faster and uses fewer parameters, while NVLM-D excels at reasoning. The 1-D tile tag is a simple, generalizable technique that could become standard in high-resolution multimodal models. Future work could explore scaling to more tiles, extending to video, or applying tile tags to other modalities like audio. NVLM sets a new bar for the community.