CogVLM2

Analysis

Why This Paper Matters

Visual language models (VLMs) have become essential for tasks ranging from OCR to video question answering, but existing models often struggle with balancing resolution, computational cost, and modality fusion. The CogVLM2 family addresses these challenges by introducing a unified architecture that scales efficiently to higher resolutions and broader modalities, including video. Its automated data generation pipeline for temporal grounding is particularly impactful, as it reduces the manual annotation burden that has historically limited video understanding research. By achieving state-of-the-art results on multiple benchmarks, this work demonstrates that careful architectural choices and training recipes can yield models that rival or surpass much larger counterparts.

How It Works

The CogVLM2 family comprises three models: CogVLM2 (image), CogVLM2-Video (video), and GLM-4V (bilingual image). All share a common architecture: a ViT encoder (EVA-CLIP) that converts images into continuous features, an adapter that bridges visual and linguistic modalities, a language model backbone (LLaMA3-8B or GLM-4-9B), and an optional visual expert module for deep fusion.

A key innovation is the adapter design. Unlike Q-former, which compresses image sequences and loses spatial details, or linear mapping, which is computationally inefficient, CogVLM2 uses a 2x2 convolutional layer followed by a SwiGLU module. This preserves image details while maintaining efficiency. The visual expert module, inherited from CogVLM, enables deep fusion of visual and linguistic features without degrading language capabilities.

For video understanding, CogVLM2-Video extracts 24 frames from input video segments and annotates each with timestamp information, allowing the language model to reason about temporal relationships. An additional 2x2 convolution layer at the end of the ViT compresses video features further.

Pre-training uses two main data strategies: iterative refinement (training on public data, then re-annotating with the model and manual correction) and synthetic data generation (e.g., rendering HTML/LaTeX to images for OCR). Key datasets include LAION-2B, COYO-700M, a digital world grounding dataset from web screenshots, and synthetic OCR datasets covering 120M English and 150M Chinese entries. The CLAY-1B dataset provides long, detailed captions generated by a fine-tuned CogVLM model.

Training recipes explore three approaches: progressively enabling more parameters, simultaneous training on language and vision data, and gradually increasing input resolution. Post-training uses a two-stage SFT: first on all VQA and alignment data to build foundational capabilities, then on a subset with preference alignment to optimize output style. For video, instruction tuning on 330k samples is followed by temporal grounding tuning on the TQA dataset.

Results

On image tasks, CogVLM2 and GLM-4V-9B achieve state-of-the-art results on OCR benchmarks (TextVQA, DocVQA, OCRbench), chart understanding (ChartQA, AI2D), and general QA (MMMU, MMVet, MMBench). They often surpass models with significantly more parameters, demonstrating the efficiency of the architecture and training recipe.

CogVLM2-Video achieves top performance on multiple video QA benchmarks, including those requiring temporal reasoning. The automated TQA data generation pipeline proves effective for grounding questions to specific video timestamps.

Significance

The CogVLM2 family represents a practical step forward in VLM design by showing that careful architectural choices—like the 2x2 conv + SwiGLU adapter and visual expert modules—can yield strong performance without scaling to enormous model sizes. The automated data generation for temporal grounding is a notable contribution that can reduce the cost of video understanding research. However, the paper does not explore failure modes or biases in synthetic data, and the models' performance on low-resource languages or niche visual domains remains untested. Overall, this work provides a solid foundation for future research in efficient, high-resolution multimodal models and automated data pipelines.