Where to show demos in your prompt?

Analysis

Why This Paper Matters

This paper addresses a critical yet overlooked aspect of in-context learning (ICL): the position of demonstrations within the prompt. While prior work has focused on demo selection and ordering, the authors show that simply moving demos from the start to the end of the user message can flip over 30% of predictions on QA tasks. This finding has immediate practical implications for anyone deploying LLMs in production, where consistency and reliability are paramount. The work also raises important questions about the reproducibility of ICL results, as small changes in prompt formatting can lead to large shifts in output.

Technical Contributions

The paper makes several key innovations:

• DPP bias definition: Formalizes the sensitivity of LLM outputs to the position of demos relative to system prompts and user messages.
• Two new metrics: ACCURACY-CHANGE measures net accuracy gain/loss from position changes, while PREDICTION-CHANGE captures output volatility (fraction of predictions that flip).
• Systematic evaluation pipeline: Tests 10 LLMs from QWEN, LLAMA3, MISTRAL, and COHERE families across four task types (classification, QA, summarization, reasoning).
• Actionable finding: Start-of-prompt demo placement consistently outperforms other positions, with gains up to +6 accuracy points.

Results

Key quantitative results:

• Start-of-prompt demos: Achieve highest accuracy and lowest prediction volatility across all tasks and models.
• End-of-user-message demos: Cause >30% prediction flips on QA tasks with no accuracy improvement.
• Model size effect: Smaller models show larger sensitivity, but even large models (e.g., 70B parameters) are marginally affected on complex reasoning tasks.
• Task sensitivity: Classification tasks are less affected than QA and reasoning, suggesting the bias is more pronounced when outputs require precise factual recall.

Significance

This work has broad implications for prompt engineering, LLM evaluation, and fairness. Practitioners can immediately adopt the recommendation to place demos at the start of prompts for more stable and accurate outputs. The findings also highlight a potential confound in existing ICL benchmarks, where demo position may have been uncontrolled. Future work should explore whether DPP bias interacts with other prompt components (e.g., chain-of-thought) and whether it extends to closed-source models. The paper sets a new standard for reporting prompt structure in ICL experiments.