Trust Before Fusion: QIMG-7 and Source-Aware Resolution for Polluted Multimodal RAG

Analysis

Why This Paper Matters

Multimodal retrieval-augmented generation (RAG) systems are increasingly deployed in real-world applications where retrieved information can be noisy, contradictory, or deliberately polluted. Naive fusion of all retrieved content—text, images, tables—can amplify errors and mislead downstream generation. This paper addresses a gap in existing benchmarks and methods by focusing on the critical step of resolving conflicts before fusion, rather than after. The introduction of QIMG-7, a dedicated dataset for polluted multimodal RAG, provides a standardized testbed that the community has lacked. This matters because as RAG systems move into high-stakes domains like medical diagnosis or legal analysis, the ability to detect and handle conflicting evidence becomes paramount.

Technical Contributions

• QIMG-7 Dataset: A curated set of multimodal queries with deliberately conflicting or polluted retrieval results, enabling systematic evaluation of robustness.
• Source-Aware Resolution Module: A lightweight component that scores each retrieved source for reliability and detects contradictions across modalities (e.g., text vs. image captions).
• Trust-Before-Fusion Pipeline: Replaces the common late-fusion approach with a pre-fusion filtering step that discards or down-weights unreliable sources.
• Evaluation Protocol: Metrics for both retrieval quality (precision/recall of reliable sources) and generation accuracy (factual consistency with ground truth).

Results

On the QIMG-7 benchmark, the proposed method achieves a 12% improvement in generation accuracy over a standard multimodal RAG baseline (late fusion of all retrieved items). When retrieval contains >30% polluted sources, the gain increases to 18%. The source-aware resolution correctly identifies 89% of conflicting pairs, compared to 62% for a naive similarity-based filter. Ablation studies confirm that the resolution step is the primary driver of improvement, not simply discarding low-confidence sources.

Significance

This paper shifts the focus from improving retrieval or generation in isolation to the often-overlooked middle step of fusion. By demonstrating that pre-fusion conflict resolution yields substantial gains, it encourages the community to design RAG pipelines that are inherently more robust to adversarial or noisy inputs. The QIMG-7 dataset will likely become a standard evaluation resource for multimodal RAG research. The broader impact lies in making AI systems that rely on retrieved knowledge more trustworthy, especially in applications where information provenance and consistency are critical.