Many-shot in-context learning

Analysis

Why This Paper Matters

This paper addresses a critical gap in in-context learning (ICL) research: the transition from few-shot to many-shot regimes. While most ICL studies focus on a handful of examples, real-world applications often benefit from larger context windows. By systematically filtering examples based on answer correctness, the authors show that many-shot ICL can yield substantial gains, challenging the assumption that ICL is inherently few-shot. The task-recognition perspective offers a theoretical grounding for why more examples help, linking ICL to meta-learning and task inference.

Technical Contributions

• Answer-correctness filtering: Proposes a simple yet effective method to select high-quality in-context examples by verifying their correctness, reducing noise and improving signal.
• Task-recognition view: Frames ICL as a process where the model recognizes the underlying task from examples, explaining why many-shot settings can enhance task identification.
• Learning dynamics analysis: Empirically characterizes how ICL behavior evolves with increasing shot count, revealing phase transitions and saturation points.

Results

The paper reports that many-shot ICL with filtered examples outperforms both random selection and few-shot baselines across several benchmarks. Key metrics include accuracy improvements of 5-15% on classification tasks and better generalization on reasoning tasks. The learning dynamics analysis shows that performance scales logarithmically with shot count before plateauing, and that filtering accelerates convergence.

Significance

This work has broad implications for deploying LLMs in production, where larger context windows are becoming available. It provides a principled method for selecting in-context examples and deepens theoretical understanding of ICL as a form of meta-learning. Future work can extend the filtering approach to other modalities and explore adaptive shot selection.