InstructBLIP

Analysis

Why This Paper Matters

Vision-language models have made remarkable progress, but most are specialized for specific tasks like captioning or VQA. The ability to follow arbitrary instructions—like a human assistant—remains a key challenge. InstructBLIP tackles this by systematically applying instruction tuning to the vision-language domain, showing that a relatively small model can outperform massive models like Flamingo-80B on zero-shot tasks. This work is a crucial step toward general-purpose multimodal AI that can adapt to diverse user requests without task-specific finetuning.

How It Works

InstructBLIP builds directly on BLIP-2, which uses a Query Transformer (Q-Former) to bridge a frozen image encoder and a frozen large language model. The key innovation is making the Q-Former instruction-aware.

The authors transform 26 publicly available datasets into instruction tuning format, covering 11 task categories including captioning, VQA, visual reasoning, and even video QA. For each task, they craft 10-15 distinct natural language instruction templates, such as "Describe this image briefly" or "Answer the question about the scene text." For datasets with short answers, they include terms like "short" to prevent the model from always generating terse outputs. OCR tokens are added as supplementary information for scene text datasets.

The core technical contribution is the instruction-aware Q-Former. In BLIP-2, the Q-Former takes only learnable query embeddings and image features. InstructBLIP additionally feeds the instruction text tokens into the Q-Former. These tokens interact with the query embeddings through self-attention layers, guiding the queries to extract visual features that are most relevant to the given instruction. The output visual vectors are then projected and fed to the frozen LLM, which generates the response.

To handle the varying sizes of training datasets, the authors sample with probabilities proportional to the square root of dataset sizes, preventing overfitting on small datasets and underfitting on large ones. They initialize from BLIP-2 checkpoints and fine-tune only the Q-Former parameters, keeping both the image encoder (ViT-g/14) and LLM frozen. Four LLM backbones are tested: FlanT5XL (3B), FlanT5-XXL (11B), Vicuna-7B, and Vicuna-13B.

Results

The results are striking. InstructBLIP achieves new state-of-the-art zero-shot performance on all 13 held-out datasets, consistently surpassing BLIP-2 by a large margin (average 15% relative improvement). The smallest InstructBLIP (FlanT5XL, 4B parameters) outperforms Flamingo-80B on all six shared evaluation datasets, with an average relative improvement of 24.8%. This demonstrates that instruction tuning is far more parameter-efficient than simply scaling model size.

Notably, InstructBLIP shows up to 47.1% relative improvement on MSRVTT-QA (video QA) over the previous SOTA, despite never being trained on temporal video data. This suggests that instruction tuning imparts a general ability to follow complex instructions that transfers to entirely new modalities.

A controlled comparison between instruction tuning and multitask learning reveals that while both perform similarly on seen datasets, instruction tuning dramatically improves zero-shot generalization on unseen tasks. This confirms that the instruction format itself, not just exposure to diverse tasks, is key to generalization.

When finetuned on downstream tasks, InstructBLIP provides a better initialization than BLIP-2, achieving SOTA on three of four datasets. The choice of LLM matters: FlanT5-based models excel at multi-choice tasks, while Vicuna-based models perform better on open-ended generation, reflecting the nature of their pretraining data.

Significance

InstructBLIP is a landmark paper that systematically demonstrates the power of instruction tuning for vision-language models. It shows that a relatively small, efficiently trained model can match or exceed massive models, and that instruction-aware feature extraction is a simple yet effective way to align visual representations with user intent. The work has inspired a wave of follow-up research on multimodal instruction tuning, including LLaVA, MiniGPT-4, and others. For practitioners, InstructBLIP provides a practical recipe: use a frozen vision encoder and LLM, fine-tune only the connector, and train on diverse instruction-formatted data. The code and models are publicly available, making this an accessible foundation for building general-purpose vision-language assistants.