OPD-V: Visual On-Policy Self-Distillation with Modality Balance

Analysis

Why This Paper Matters

Multimodal large language models (MLLMs) have become central to AI systems that need to reason across text and images. However, a persistent issue is modality imbalance, where the model relies too heavily on textual cues and underutilizes visual information. This paper tackles this problem in the context of on-policy self-distillation (OPSD), a common post-training technique. The authors argue that existing OPSD methods overlook modality imbalance, leading to underused privileged information and suboptimal reasoning.

The paper's significance lies in its novel insight: modality balance itself can serve as privileged information. By constructing a Positive Teacher (with zoomed-in images) and a Negative Teacher (with masked images), they show that the degree of modality imbalance directly affects reasoning correctness and token logits. This finding motivates a new paradigm, OPD-V, which uses modality-balance trust regions to guide self-distillation. This is a fresh angle that could inspire further research on balancing modalities in MLLMs.

Technical Contributions

• Identification of Modality Imbalance in OPSD: The paper systematically analyzes how textual dominance limits the effectiveness of privileged information in self-distillation.
• Positive and Negative Teacher Design: Introduces two teacher variants (Zoom-In and Mask) to create different degrees of modality imbalance, providing a controlled way to study the phenomenon.
• Modality-Balance Logits Margins: A novel metric that quantifies the balance between modalities in the logits, used to define a trust region.
• Modality-Balance Trust Region: A selection mechanism that filters on-policy tokens for self-distillation, ensuring only tokens with balanced modality influence are used, improving training efficiency.
• Comprehensive Evaluation: Validates OPD-V across 6 benchmarks, 4 MLLM backbones, and 5 post-training methods, showing consistent gains.

Results

The paper reports that OPD-V consistently improves reasoning performance across all tested benchmarks and backbones. For instance, it outperforms existing OPSD methods while also reducing training cost. The exact metrics are not detailed in the abstract, but the claim of consistent improvement across 6 benchmarks and 4 backbones suggests strong generalizability. The reduction in training cost is a practical advantage, making OPD-V attractive for real-world deployment.

Significance

OPD-V offers a new direction for improving MLLM reasoning by explicitly addressing modality imbalance. This could lead to more robust multimodal systems that fully leverage visual information, which is crucial for applications like autonomous driving, medical imaging, and robotics. The concept of using modality balance as privileged information may also extend to other modalities (e.g., audio, video) and other self-distillation frameworks. The paper's findings encourage the community to consider modality balance as a first-class design principle in MLLM training.