Conference Paper
Large Language Models
Featured

Language Models are Few-Shot Learners

Tom B. Brown(OpenAI), Benjamin Mann(OpenAI), Nick Ryder(OpenAI), Melanie Subbiah(OpenAI), Jared Kaplan(Johns Hopkins University), Prafulla Dhariwal(OpenAI), Arvind Neelakantan(OpenAI), Pranav Shyam(OpenAI), Girish Sastry(OpenAI), Amanda Askell(OpenAI)
May 28, 2020NeurIPS 202028,000 citations
arXivPDFDOI

28k

Citations

3.5k

Influential Citations

NeurIPS 2020

Venue

2020

Year

Abstract

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions — something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting.

Analysis

Why This Paper Matters

GPT-3 demonstrated that scaling language models to 175 billion parameters unlocked a qualitatively different capability: in-context learning. Rather than fine-tuning on thousands of labeled examples, GPT-3 could perform new tasks from just a few demonstrations provided in the prompt. This insight launched the era of prompt engineering and established that scale itself could be a path to general-purpose AI capabilities.

Technical Contributions

  • 175 billion parameter model: 10x larger than any previous dense language model, trained on a filtered version of Common Crawl plus curated datasets (300 billion tokens).
  • In-context learning paradigm: Systematic evaluation of zero-shot, one-shot, and few-shot performance across dozens of NLP benchmarks — without any gradient updates at inference time.
  • Scaling laws empirically validated: Performance on nearly every benchmark improved smoothly and predictably with model scale.

Results

GPT-3 achieved strong results across a wide range of tasks:

  • Translation: Competitive with supervised systems on some language pairs
  • Question answering: Near state-of-the-art on TriviaQA in zero-shot
  • Arithmetic: Able to perform two-digit addition and subtraction
  • News article generation: Human evaluators had difficulty distinguishing GPT-3 articles from real ones

Legacy

GPT-3 launched the modern era of large language models and demonstrated that prompting could replace fine-tuning for many applications. It directly led to ChatGPT, the AI-powered coding assistant wave, and the competitive scaling race among major AI labs.

gpt-3few-shotin-context-learningscalinglanguage-modelopenainlp
Large Language ModelsFew Shot LearningIn Context LearningScaling Laws

Related Papers