Flan T5, Flan PaLM

Analysis

Why This Paper Matters

Instruction fine-tuning has become a cornerstone of modern language model development, but its scaling properties were poorly understood. This paper systematically investigates how increasing the number of tasks, model size, and the inclusion of chain-of-thought (CoT) data affect performance. The findings are critical for practitioners deciding how to allocate compute between pre-training and fine-tuning, and for understanding the limits of multi-task learning.

How It Works

The authors curate a massive fine-tuning dataset comprising 473 datasets, 146 task categories, and 1,836 total tasks. They apply a variety of instruction templates to each dataset, ensuring diversity in how tasks are presented.

The fine-tuning procedure, called Flan (Fine-tuning Language Models), is applied to T5 and PaLM models of varying sizes (8B, 62B, 540B). The key innovation is the inclusion of nine datasets with chain-of-thought annotations, which teach the model to generate intermediate reasoning steps before producing an answer.

The authors evaluate on held-out benchmarks including MMLU, BBH, GSM8K, and TyDiQA, comparing Flan-tuned models against their non-finetuned counterparts and prior state-of-the-art systems.

Results

Multi-task instruction fine-tuning yields consistent improvements of 9.4% to 15.5% across model sizes. However, the benefit of adding more tasks saturates after about 282 tasks, suggesting that task diversity matters more than sheer quantity.

Including CoT data in the fine-tuning mixture significantly boosts reasoning performance. Flan-PaLM 540B achieves 75.2% on MMLU and 83.9% on GSM8K, both new state-of-the-art results at the time. CoT prompting combined with self-consistency further improves multilingual math reasoning on MGSM.

Importantly, fine-tuning on both CoT and non-CoT data does not hurt non-CoT performance, while fine-tuning on only non-CoT data degrades CoT reasoning. This indicates that CoT examples are essential for maintaining reasoning abilities.

Significance

This paper provides a practical recipe for instruction fine-tuning: use a diverse set of tasks (around 300), include chain-of-thought data, and scale model size. The compute cost of fine-tuning is small relative to pre-training, making it an efficient way to boost performance.

The combination of instruction fine-tuning with UL2 continued pre-training yields the strongest model, showing that these methods are complementary. This work has influenced subsequent large language model development, including the Flan-T5 and Flan-PaLM series, and underscores the importance of careful data curation and task diversity in fine-tuning.