Doc Former

Analysis

Why This Paper Matters

Document understanding is a critical task for automating workflows in finance, legal, and healthcare sectors. Traditional approaches often treat text and layout separately, or fuse modalities only at the final layer, missing the rich interplay between visual cues (like logos, signatures, and handwriting) and textual content. DocFormer addresses this gap by proposing a transformer architecture that deeply integrates visual, textual, and spatial information from the start. Its discrete multi-modal design—where each modality retains its own representation while sharing spatial features—offers a principled way to combine heterogeneous data types without forcing unnatural concatenation or late fusion. This matters because it achieves superior performance on standard benchmarks, showing that careful architectural choices can unlock better understanding of complex documents.

How It Works

DocFormer is an encoder-only transformer with a CNN backbone for visual feature extraction, trained end-to-end. The architecture is built around three key feature streams: visual, language, and spatial.

Visual Features: A ResNet50 extracts a low-resolution feature map from the document image. A 1x1 convolution reduces channel dimensions, then a linear layer flattens the map into a sequence of 512 tokens, each of dimension 768.

Language Features: Text is tokenized using a word-piece tokenizer and passed through a trainable embedding layer initialized with LayoutLMv1 weights, producing a sequence of the same shape as the visual tokens.

Spatial Features: For each word, bounding box coordinates are encoded using separate layers for x and y coordinates. Additional spatial cues—height, width, distances between bounding box corners and centroids—are computed and summed with absolute 1D positional encodings. Separate spatial embeddings are created for visual and language features, acknowledging that spatial dependencies may differ between modalities.

Multi-Modal Self-Attention Layer: This is the core innovation. Unlike standard self-attention that processes a single sequence, DocFormer computes attention separately for visual and language tokens, but both share the same spatial features. The attention formulation incorporates an inductive bias that local features are more important than global ones for document understanding. The output of each transformer layer is the sum of the spatially-aware self-attended visual and language features, producing a multi-modal representation.

Pre-training Tasks: Three tasks are designed to encourage cross-modal collaboration:

• Multi-Modal Masked Language Modeling (MMMLM): Masks text tokens but does not mask corresponding visual regions, forcing the model to use visual context to reconstruct text.
• Learn To Reconstruct (LTR): An auto-encoder style task that reconstructs the document image from multi-modal features, requiring both modalities to work together.
• Text Describes Image (TDI): A binary classification task that determines whether a given text matches the document image, with 20% of batches containing mismatched pairs.

Results

DocFormer is evaluated on four benchmarks: FUNSD (form understanding), Kleister-NDA (receipt understanding), CORD (receipt extraction), and RVL-CDIP (document image classification). It achieves state-of-the-art results on all four, outperforming prior methods like LayoutLMv2 and BROS. The gains are particularly notable on FUNSD and CORD, where the deep multi-modal fusion helps in accurately extracting key-value pairs from complex layouts.

Significance

DocFormer demonstrates that discrete, deeply integrated multi-modal features are more effective than joint concatenation or late fusion for document understanding. Its architecture provides a template for future work: separate modality-specific streams that interact through shared spatial features and a carefully designed attention mechanism. The pre-training tasks are also noteworthy—by not masking visual regions during language modeling, the model learns to rely on visual context, which is a natural inductive bias for documents where layout and appearance carry meaning. This work has practical implications for automating document processing in industries like banking, insurance, and legal, where accurate extraction of information from forms and receipts is essential. Future research could extend DocFormer to handle more diverse document types, incorporate language models with larger capacities, or explore unsupervised spatial feature learning to reduce reliance on bounding box annotations.