FLAN

Analysis

Why This Paper Matters

Large language models (LLMs) like GPT-3 have shown remarkable few-shot learning abilities through in-context prompting, but their zero-shot performance often lags behind. The FLAN paper introduces a straightforward yet highly effective technique—instruction tuning—that bridges this gap. By fine-tuning a pretrained model on a diverse collection of tasks described via natural language instructions, the model learns to generalize to unseen tasks without any examples. This work is pivotal because it demonstrates that with proper training, LLMs can become more instruction-following and user-friendly, reducing the need for task-specific fine-tuning or complex prompt engineering.

How It Works

FLAN (Fine-tuned LAnguage Net) starts with LaMDA-PT, a 137B parameter decoder-only transformer pretrained on a large corpus of web documents, dialog data, and Wikipedia. The core innovation is instruction tuning: the model is fine-tuned on a mixture of 62 publicly available NLP datasets, each converted into natural language instruction templates. For every dataset, the authors manually create 10 unique templates that describe the task in different ways, including up to three "turned around" templates that invert the task (e.g., given a hypothesis, generate a premise).

Datasets are grouped into 12 task clusters (e.g., natural language inference, reading comprehension, translation). To evaluate zero-shot performance on unseen tasks, the authors adopt a rigorous hold-out strategy: for each evaluation cluster, they train a separate model on all other clusters, ensuring that no task type from the held-out cluster appears during instruction tuning. This conservative evaluation prevents any accidental exposure to similar tasks.

During instruction tuning, the model is trained with input sequences up to 1024 tokens and target sequences up to 256 tokens. To balance dataset sizes, each dataset contributes at most 30k examples, and mixing rates are capped at 3k. The training mixes all datasets and randomly samples from them, ensuring diverse exposure.

Results

FLAN is evaluated on a wide range of unseen task types, including natural language inference (NLI), reading comprehension, closed-book QA, translation, commonsense reasoning, coreference resolution, and struct-to-text. The results are striking:

• NLI: FLAN outperforms GPT-3 by a large margin on all five NLI datasets. By phrasing NLI as the more natural question "Does <premise> mean that <hypothesis>?", FLAN achieves much higher accuracy.
• Reading comprehension: FLAN surpasses GPT-3 on BoolQ, MultiRC, and OBQA, with particularly strong gains on BoolQ.
• Closed-book QA: FLAN outperforms GPT-3 on all four datasets (ARC-easy, ARC-challenge, NQ, TriviaQA) and compares favorably with GLaM.
• Translation: FLAN outperforms zero-shot GPT-3 on all six translation directions (French-English, German-English, Romanian-English) but lags behind few-shot GPT-3 in most cases. It excels at translating into English but is weaker for English-to-other-language translation, likely due to the English-centric tokenizer and pretraining data.

However, instruction tuning does not improve performance on tasks that are similar to the original language modeling objective, such as commonsense reasoning and coreference resolution when formulated as sentence completions. For seven such tasks, FLAN only outperforms LaMDA-PT on three, indicating that instruction tuning is not universally beneficial.

Significance

FLAN demonstrates that instruction tuning is a simple, scalable, and effective method for improving zero-shot learning in large language models. By training on a diverse set of tasks described via natural language instructions, the model learns to generalize to unseen tasks without any examples. This work has significant practical implications: it reduces the need for task-specific fine-tuning or complex prompt engineering, making LLMs more accessible and versatile for real-world applications. The paper also highlights the importance of task diversity and template design in instruction tuning, and it identifies limitations where instruction tuning does not help, guiding future research. FLAN has inspired subsequent work on instruction-tuned models like T0 and InstructGPT, cementing its influence on the trajectory of LLM development.