LongLLMLingua

Analysis

Why This Paper Matters

Large language models (LLMs) struggle with long contexts due to quadratic attention costs, latency, and performance degradation—especially when relevant information is buried in the middle. LongLLMLingua tackles these issues head-on by compressing prompts intelligently, making LLMs both faster and more accurate in scenarios like multi-document QA or long-form reasoning. This matters for practitioners who need to balance cost, speed, and quality in production systems.

How It Works

The framework builds on LLMLingua but introduces question-awareness and document reordering. The core idea is to maximize the density of question-relevant information in the compressed prompt.

Question-Aware Coarse-to-Fine Compression operates in two stages. First, coarse-grained compression ranks documents by a metric (r_k) based on document-level perplexity when conditioned on the question. Documents with lower perplexity (better predicted by the model) are deemed more relevant. This is computed as:

where (x_{que,restrict i}) is the i-th token in the concatenation of the question and a restriction phrase. Then, fine-grained compression uses contrastive perplexity—the distribution shift caused by the question—to identify tokens strongly associated with the query, rather than just high-perplexity tokens that may be irrelevant.

Document Reordering addresses the well-known "lost-in-the-middle" phenomenon. Documents are sorted by their importance scores so that the most relevant ones appear at the beginning or end of the prompt, where LLMs perform best.

Dynamic Compression Ratios bridge coarse and fine compression. The budget for each document is allocated adaptively based on its relevance rank, using a linear scheduler:

where (N_d) is the number of documents and (\delta_\tau) controls overall budget.

Post-Compression Subsequence Recovery restores critical tokens that may have been dropped during token-level compression, leveraging subsequence relationships in the original prompt, compressed prompt, and LLM response.

Results

LongLLMLingua was evaluated on NaturalQuestions, LongBench, and ZeroSCROLLS using GPT-3.5-Turbo and LongChat-13B-16k, with LLaMA-2-7B-Chat as the compression model. It consistently outperformed retrieval-based methods (BM25, Sentence-BERT) and compression baselines (Selective Context, LLMLingua) across compression ratios.

Latency was significantly reduced, especially at higher compression rates, making it suitable for real-time applications.

Ablation studies confirmed that each component—question-aware compression, dynamic ratios, and subsequence recovery—contributes to overall performance.

Significance

LongLLMLingua offers a practical, modular framework for prompt compression that directly addresses real-world deployment challenges. By making LLMs more efficient in long-context scenarios, it reduces API costs and inference latency while maintaining or improving accuracy. The question-aware design ensures that compression preserves task-relevant information, and the reordering strategy mitigates a known weakness of transformer models. This work is a step toward more scalable and cost-effective LLM applications in document analysis, customer support, and other domains requiring processing of lengthy inputs.