Aya Vision

Analysis

Why This Paper Matters

Building vision-language models that work across many languages is a critical step toward truly global AI. Most VLMs today are English-centric, leaving speakers of other languages with limited or no access to multimodal capabilities. Aya Vision tackles this head-on by creating open-weight models supporting 23 languages, from Arabic to Vietnamese. The paper's core insight is that the bottleneck isn't just data volume—it's data quality and the risk of catastrophic forgetting when adding vision to a text-only model. By introducing a synthetic annotation pipeline and a cross-modal merging technique, the authors show it's possible to achieve state-of-the-art multilingual performance without sacrificing the language skills the backbone model already has.

How It Works

The approach rests on two main pillars: a data framework for generating high-quality multilingual multimodal instruction data, and a model merging strategy to preserve text-only performance.

Data Framework

The synthetic annotation pipeline has three stages. First, distillation-based recaptioning uses a teacher model to rewrite captions for existing datasets (Cauldron, PixMo, etc.) into more detailed, natural, and diverse versions while staying anchored to ground-truth answers. This increases average word count from 14.2 to 100.1 and lexical diversity (MTLD) from 11.0 to 61.2. Second, a two-stage filtering pipeline removes low-quality samples: keyword detection catches refusals and repetitions, then a semantic judge (command-r-plus) verifies that the rephrased caption still answers the original question correctly, discarding 3.2% of samples. Third, a hybrid translation pipeline uses NLLB-3.3B for initial machine translation into 22 languages, followed by LLM-based post-editing to correct artifacts and improve fluency.

The final training mixture balances three categories: synthetically re-annotated English data (2.29M samples, upsampled to 3.5M), multilingual versions of that data (3.4M samples sampled uniformly across 22 languages), and high-quality original datasets (3.7M samples, downsampled to avoid degrading conversational quality). About 66% of training data is synthetically re-annotated, with 35% being multilingual.

Model Merging

To prevent forgetting text-only skills, the authors perform linear interpolation between the text-only LLM (Aya Expanse) and the multimodal backbone after vision-language training. The vision encoder and alignment layer are inherited from the VLM. This simple technique allows the model to retain its original language capabilities while gaining strong multimodal performance.

Architecture

Both models use SigLIP2 vision encoders (patch14-384 for 8B, patch16-512 for 32B). Images are split into up to 12 non-overlapping tiles plus a thumbnail for low-resolution context. A 2-layer SwiGLU MLP connector with Pixel Shuffle reduces image tokens by 4×, yielding max 2,197 tokens for 8B and 3,328 for 32B. The language model for 8B is Command-R7B post-trained with Aya Expanse; for 32B it's Aya-Expanse-32B.

Training happens in two steps: vision-language alignment (training only the connector, with 14% multilingual data added to LLaVa-Pretrain), then supervised fine-tuning (training connector and LLM, freezing vision encoder). Sequence packing with length 8192 improves efficiency.

Results

Aya-Vision-8B achieves best-in-class win-rates on AyaVisionBench and m-WildVision, ranging from 49.6% to 80.3% against other models. It outperforms Qwen-2.5-VL-7B and Pixtral-12B by 54.8% average win-rate, and surpasses Gemini-Flash1.5-8B with 60.3% win-rate. Against Pangea-7B, which was trained on extensive multilingual data, Aya-Vision-8B achieves a 71.7% win-rate. In English, it remains competitive with Molmo-7B (48.3% win-rate) while outperforming it in other languages by 80%.

Aya-Vision-32B consistently beats models over 2× larger, including Molmo-72B, Qwen-2.5-VL-72B, and Llama-3.2-90B-Vision, with win-rates from 48.5% to 73%. Against Llama-3.2-90B-Vision, it achieves 65.9% and 73% win-rates on the two benchmarks.

Significance

Aya Vision demonstrates that open-weight models can compete with—and often beat—much larger proprietary systems in multilingual multimodal tasks. The synthetic annotation pipeline and cross-modal merging are practical techniques that other researchers can adopt. By releasing models in two sizes (8B and 32B), the work makes advanced multimodal AI accessible on consumer hardware while still offering a high-performance option. The main limitation is reliance on synthetic data, which may introduce subtle biases, and the 8B model's slightly lower performance on the more challenging AyaVisionBench. Nevertheless, Aya Vision sets a new standard for multilingual VLMs and provides a strong foundation for future work in inclusive AI.