Vision-language models for medical report generation and visual question answering: a review

Analysis

Why This Paper Matters

Medical vision-language models (VLMs) are a rapidly growing area at the intersection of computer vision and natural language processing, with direct applications in clinical decision support. This review is timely because the field has seen a surge of models and datasets, yet lacks a consolidated overview. By cataloging 18 datasets and 16 models, the paper provides a valuable map for researchers entering the domain or seeking to benchmark their work. The explicit focus on publicly available resources makes the review actionable for practitioners who cannot access proprietary medical data.

Moreover, the paper highlights persistent challenges—limited annotated data, privacy constraints, and the absence of standardized evaluation metrics—that hinder clinical translation. These issues are not merely technical but also regulatory and ethical, making the review relevant beyond AI research. The proposed future directions, such as federated learning and synthetic data generation, offer concrete pathways for the community.

Technical Contributions

• Comprehensive dataset survey: The paper reviews 18 public medical vision-language datasets, covering modalities like chest X-rays, pathology slides, and retinal images. This is the first such consolidated list in a review format.
• Model taxonomy: It categorizes 16 recent medical VLMs (e.g., BioViL, CheXzero, MedBLIP) by architecture (e.g., dual-encoder vs. fusion encoder) and pre-training strategy (e.g., contrastive learning, masked language modeling).
• Evaluation metric analysis: The authors discuss metrics for report generation (e.g., BLEU, ROUGE, CIDEr) and VQA (e.g., accuracy, F1), noting their limitations in capturing clinical correctness.
• Challenge identification: Key obstacles are systematically enumerated: data scarcity, privacy laws (HIPAA, GDPR), lack of domain-specific metrics, and model interpretability.

Results

The review does not present new experimental results but synthesizes findings from the literature. It notes that Transformer-based architectures are the dominant approach for fusing visual and textual modalities. For report generation, models like BioViL and CheXzero achieve strong BLEU scores on the MIMIC-CXR dataset, but clinical accuracy remains suboptimal. For VQA, models such as MedBLIP and PubMedCLIP show competitive accuracy on the VQA-RAD dataset, yet performance drops on out-of-distribution questions. The paper emphasizes that no single metric reliably captures clinical utility, and many models are evaluated on different splits, making direct comparison difficult.

Significance

This review provides a structured foundation for the medical VLM community. By organizing datasets, models, and metrics, it lowers the barrier to entry for new researchers and helps practitioners select appropriate resources. The explicit discussion of privacy and evaluation challenges pushes the field toward more rigorous and clinically meaningful benchmarks. As healthcare AI moves toward multimodal understanding, this paper serves as a critical reference point for future work on robust, privacy-preserving, and interpretable medical VLMs.