Zero-Shot Reasoners (2022)
Free"Let's think step by step" — zero-shot CoT milestone
FreeFree tier
About Zero-Shot Reasoners (2022)
Zero-Shot Reasoners is a research paper from NeurIPS 2022 that introduces Zero-shot-CoT, a simple prompting technique for large language models (LLMs). By appending the phrase "Let's think step by step" to each prompt, the method elicits chain-of-thought reasoning without requiring any few-shot examples. Experiments show substantial accuracy gains across arithmetic (e.g., MultiArith: 17.7% to 78.7%; GSM8K: 10.4% to 40.7% with InstructGPT), symbolic reasoning, and logical reasoning tasks, and similar improvements with a 540B parameter PaLM model. The work demonstrates that LLMs possess untapped zero-shot reasoning capabilities and provides a strong baseline for reasoning benchmarks.
Key Features
Zero-shot chain-of-thought (Zero-shot-CoT) prompting
Single prompt template: "Let's think step by step"
No hand-crafted few-shot examples required
Works with multiple LLMs including InstructGPT and PaLM
Improves accuracy on arithmetic, symbolic, and logical reasoning tasks
Accepted at NeurIPS 2022
Code publicly available on GitHub
Pros & Cons
Pros
- Simple and effective: one prompt template works across diverse tasks
- Eliminates the need for task-specific few-shot examples
- Substantial accuracy improvements over standard zero-shot prompting
- Applicable to large-scale models without additional training or fine-tuning
Cons
- Performance still lags behind few-shot chain-of-thought prompting on some tasks
- Effectiveness may vary across different LLMs and prompt formulations
Best For
Arithmetic reasoning (MultiArith, GSM8K, AQUA-RAT, SVAMP)Symbolic reasoning (Last Letter, Coin Flip)Logical reasoning (Date Understanding, Tracking Shuffled Objects)Establishing a strong zero-shot baseline for reasoning benchmarks
FAQ
What is Zero-shot-CoT?
Zero-shot-CoT is a prompting method that adds the phrase "Let's think step by step" before each answer, enabling large language models to perform chain-of-thought reasoning without any few-shot examples.
Which models does Zero-shot-CoT work with?
The method was tested on InstructGPT (text-davinci-002) and PaLM (540B parameters), showing similar magnitudes of improvement across both models.
What type of tasks does Zero-shot-CoT improve?
The method improves performance on arithmetic (e.g., MultiArith, GSM8K), symbolic reasoning (e.g., Last Letter, Coin Flip), and logical reasoning tasks (e.g., Date Understanding, Tracking Shuffled Objects).
Is Zero-shot-CoT better than few-shot CoT?
Zero-shot-CoT is competitive but generally still lags behind few-shot CoT which uses task-specific exemplars. However, it dramatically outperforms standard zero-shot prompting and eliminates the need for hand-crafted examples.