Aya 101

Analysis

Why This Paper Matters

Most large language models today are trained on a handful of high-resource languages like English, Chinese, and Spanish. This leaves billions of speakers of lower-resourced languages without access to capable AI assistants. Aya 101 directly tackles this gap by creating a single model that follows instructions in 101 languages—over half of which are considered lower-resourced. It not only covers more languages than prior models like mT0 and BLOOMZ but also outperforms them on most tasks. This is a significant step toward democratizing AI across the world's linguistic diversity.

How It Works

Aya 101 is built by fine-tuning mT5, a 13-billion-parameter encoder-decoder transformer that was pre-trained on 1 trillion tokens covering 101 languages from mC4. The core innovation lies in the instruction fine-tuning (IFT) data mixture, which is carefully curated from multiple sources to ensure quality, diversity, and balance.

Data Sources and Curation

The training data comes from four main sources:

1. Multilingual Templates: The xP3x dataset (680M examples across 277 languages) is pruned to 101 languages. Two reviewers inspect each template, removing those with very short or empty generations, near-duplicate prompts, or grammatical errors. This pruning removes 50.2% of English and 35.9% of multilingual templates, but increases average instruction length by 7.0% for English and 16.8% for multilingual instances.

1. Human Annotations: The Aya Dataset provides new multilingual human annotations curated by fluent speakers.

2. Machine Translation: 19 translated datasets covering 101 languages are included, with up to 3,000 instances randomly sampled per language per dataset to preserve diversity. The final translated mixture contains 7.5M instances.

3. Synthetic Data: ShareGPT-Command is a 6.8M synthetically generated and machine-translated dataset in 93 languages. It combines human-annotated prompts from ShareGPT with synthetic English completions from Cohere's Command model, then translates them using NLLB.

Languages are grouped into five clusters based on resource availability: lower-resourced (LR), mid-resourced (MR), and higher-resourced (HR).

Training Strategy

The combined sources total over 203M instances, but volumes are highly skewed—human annotations make up only 0.7% of the budget. To handle this, a two-fold sampling strategy is used: source-level sampling and dataset-level sampling. Three weighting schemes are explored:

• Human Annotation Heavy: upweights the Aya Dataset
• Translation Heavy: upweights translated sources
• Template Heavy: upweights xP3x and Data Provenance

All models are trained for 30,000 update steps (25M samples). A safety-mitigated variant, Aya Safe, is also trained using adversarial prompts from multilingual AdvBench and XSafety benchmarks.

Results

Discriminative Tasks

On unseen tasks (XWinograd, XNLI, XCOPA, XStoryCloze), the Aya model (template-heavy) achieves 75.12% average performance, outperforming mT0 (72.9%), BLOOMZ, and Bactrian-X. Notably, mT0x—which also covers 101 languages—performs significantly worse than mT0, showing that data quality matters more than model capacity.

On multilingual MMLU (26 languages), Aya (translation-heavy) improves accuracy by 21.1% over mT0x, 18.4% over mT0, and 25.1% over Okapi.

Generative Tasks

On FLORES-200 machine translation, Aya (translation-heavy) achieves spBLEU scores of 44% (X→English) and 31% (English→X). On XLSum summarization and Tydi-QA question answering, gains are more modest (1.8% RougeLsum and 2.2% F1), likely due to limited language coverage in those datasets.

Performance by Language Resourcedness

Aya shows the largest improvements over mT0x in high-resource languages for discriminative tasks (12.1% on unseen tasks, 21.8% on MMLU) and in low-resource languages for machine translation (47.1% spBLEU improvement). This demonstrates that the balanced data mixture particularly benefits lower-resourced languages.

Significance

Aya 101 demonstrates that careful data curation—not just model size or architecture—is the key to building capable multilingual models. By combining pruned templates, human annotations, machine translation, and synthetic data with a balanced sampling strategy, the team achieves state-of-the-art performance across 101 languages. The open-access release on Hugging Face enables researchers and practitioners worldwide to build on this work, potentially accelerating AI adoption in underserved linguistic communities. The safety-mitigated variant also addresses the critical need for responsible deployment in multilingual settings.