BLOOMZ, mT0

Analysis

Why This Paper Matters

Large language models are predominantly trained on English data, limiting their utility for speakers of other languages. This paper tackles a critical question: can we make these models generalize to non-English languages without explicit multilingual fine-tuning? The answer is a resounding yes, and the implications are profound for democratizing AI across linguistic boundaries. By showing that fine-tuning on English tasks alone can unlock zero-shot performance in dozens of languages, the authors provide a practical path to multilingual capability without the prohibitive cost of collecting labeled data in every target language.

How It Works

The approach is elegantly simple: take a pretrained multilingual model (BLOOM or mT5) and fine-tune it on a diverse set of supervised tasks, all prompted in English. The key insight is that the model's pretraining already exposed it to many languages; fine-tuning on English tasks teaches it to follow instructions, and this skill transfers to languages it has seen before.

The authors create xP3, a dataset that extends the English-only P3 with multilingual tasks in 46 languages. They add four new task clusters: translation, simplification, program synthesis, and miscellaneous code datasets.

To study the effect of prompt language, they also create xP3mt, where prompts are machine-translated into the dataset's language. This allows them to compare English-only prompts against multilingual prompts.

The models are fine-tuned in three variants: BLOOMZ-P3 (English-only P3), BLOOMZ (xP3 with English prompts), and BLOOMZ-MT (xP3mt with machine-translated prompts). The same variants are created for mT5 (mT0, mT0-P3, mT0-MT).

Results

The results are striking. Fine-tuned models dramatically outperform their pretrained counterparts on held-out tasks across languages.

A key finding is that mT0-13B (encoder-decoder) beats BLOOMZ-176B (decoder-only) despite having 10x fewer parameters. This suggests that the encoder-decoder architecture and masked language modeling pretraining are more conducive to cross-lingual generalization.

Interestingly, fine-tuning on xP3 (which is only 39% English) actually improves performance on English tasks compared to fine-tuning on 100% English P3. This indicates that multilingual data helps even for English-only evaluation.

However, the paper also reveals limitations. Multitask fine-tuning degrades performance on generative tasks like code generation, and models still prefer English prompts over non-English ones.

Significance

This paper provides a practical recipe for building multilingual AI systems: pretrain on diverse languages, then fine-tune on English tasks. The open-source release of models and datasets lowers the barrier for researchers and practitioners to replicate and extend these results. The finding that encoder-decoder architectures are more sample-efficient for cross-lingual tasks has implications for model design. While generative tasks remain challenging, the paper's core insight—that task generalization can cross language boundaries—is a major step toward truly multilingual AI.