Longformer

Analysis

Why This Paper Matters

Standard Transformer models suffer from quadratic self-attention complexity, limiting their input length to typically 512 tokens. This is a critical bottleneck for tasks like document classification, question answering over long articles, or summarization of entire books. Longformer directly tackles this by proposing a sparse attention pattern that scales linearly with sequence length, making it feasible to process documents with tens of thousands of tokens without sacrificing performance.

How It Works

Longformer replaces the full self-attention matrix with a combination of three attention patterns:

1. Sliding Window Attention: Each token attends only to a fixed-size window of neighboring tokens (e.g., 512 tokens). Multiple stacked layers create a large effective receptive field, similar to CNNs.

2. Dilated Sliding Window: To further expand the receptive field without increasing computation, gaps are introduced in the window (dilation rate d). This allows tokens to attend to farther-away positions with fewer operations.

1. Global Attention: A small number of pre-selected tokens (e.g., [CLS] tokens or question tokens) attend to all tokens in the sequence, and all tokens attend to them. This symmetric global attention ensures that critical information can propagate across the entire input.

To implement this efficiently, Longformer uses two separate sets of linear projections: one for sliding window attention (Qs, Ks, Vs) and one for global attention (Qg, Kg, Vg). The global projections are initialized from the local ones, allowing the model to specialize each attention type during fine-tuning.

For autoregressive language modeling, the authors use dilated sliding window attention with varying window sizes across layers: small windows in lower layers capture local patterns, while larger windows in higher layers build global representations. Lower layers avoid dilation to maximize local context learning.

Training proceeds in phases: starting with short sequences (2,048 tokens) and small windows, then doubling both sequence length and window size while halving the learning rate across five phases, ending with sequences of 23,040 tokens. This staged approach keeps training efficient.

Results

Longformer is evaluated on long-document tasks using sequences up to 32,256 tokens. It achieves state-of-the-art results on several benchmarks, including coreference resolution and question answering, while using significantly less memory and compute than full-attention baselines. The model is pretrained by continuing from RoBERTa with masked language modeling on a corpus of long documents, adding position embeddings up to 4,096 tokens. A variant with frozen RoBERTa weights (only training new position embeddings) also performs well, demonstrating the plug-and-play nature of the attention pattern.

For sequence-to-sequence tasks, Longformer-Encoder-Decoder (LED) is introduced, initialized from BART. The encoder uses the efficient local+global attention, while the decoder uses full self-attention over encoded tokens and previously decoded positions. Position embeddings are extended to 16K tokens.

Significance

Longformer was a landmark paper that made long-context transformers practical. Its sparse attention pattern inspired many subsequent works (e.g., BigBird, Sparse Transformers) and enabled applications like long-document summarization, legal document analysis, and scientific paper understanding. By showing that pretrained models can be adapted with minimal changes, Longformer lowered the barrier for practitioners to handle longer sequences. The paper's clear design choices—sliding window, dilation, and global attention—remain a standard reference point for efficient transformer architectures.