Chain of Thought Empowers Transformers to Solve Inherently Serial Problems
FreeProvable expressiveness boost from chain-of-thought reasoning in transformers
About Chain of Thought Empowers Transformers to Solve Inherently Serial Problems
Chain of Thought Empowers Transformers to Solve Inherently Serial Problems is a theoretical research paper accepted at ICLR 2024. It provides a formal analysis of how chain-of-thought (CoT) reasoning enhances the expressive power of decoder-only transformers. The paper proves that constant-depth transformers with constant-bit precision can only solve problems in AC0 without CoT, but with T steps of CoT can solve any problem solvable by boolean circuits of size T. Empirically, CoT dramatically improves accuracy on tasks that are hard for parallel computation, including composition of permutation groups, iterated squaring, and circuit value problems, especially for low-depth transformers.
Key Features
Pros & Cons
- Provides rigorous theoretical foundation for chain-of-thought benefits
- Shows CoT dramatically improves accuracy on serial reasoning tasks
- Offers insights into transformer expressiveness limitations and how to overcome them
- Findings are theoretical and limited to constant-depth, constant-bit precision transformers
- Empirical validation focuses on synthetic tasks rather than real-world applications
- Requires explicit generation of intermediate steps, which may increase inference cost