Document Screenshot Embedding

Analysis

Why This Paper Matters

Traditional document retrieval systems rely on extracting text content—via OCR or HTML parsing—before indexing. This approach loses visual information (images, charts, layout) and struggles with diverse formats like slides, PDFs, or scanned documents. Document Screenshot Embedding (DSE) flips the paradigm: instead of extracting content, it directly encodes a screenshot of the entire document into a dense vector. By leveraging a large vision-language model (Phi-3-vision), DSE preserves all information—text, images, and layout—in a single representation, enabling unified retrieval across formats without any preprocessing.

How It Works

DSE adopts a bi-encoder architecture. A document screenshot is fed into a vision encoder (Phi-3-vision), while a text query is encoded separately. The similarity between the two embeddings is computed via cosine similarity.

The key innovation is replacing a standard CLIP vision encoder with Phi-3-vision, which can represent an image with more patches by cropping it into sub-images. A screenshot is resized to (Cx * 336) x (Cy * 336) pixels and divided into Cx * Cy sub-images, each 336x336 pixels. Each sub-image is encoded into 576 patch latent representations. An additional global 336x336 encoding captures overall context. This yields (Cx * Cy + 1) * 576 patches. Every four patches are concatenated and projected into one embedding for the language model, resulting in (Cx * Cy + 1) * 576 / 4 input embeddings.

These patch embeddings are concatenated with a text prompt: "<s><img> What is shown in this image?</s>". The embedding of the end-of-sequence token from the last hidden state serves as the document screenshot embedding.

Training uses contrastive learning with the InfoNCE loss, where the similarity between query and document is cosine similarity. Hard negatives and in-batch negatives are used to improve discrimination.

Results

In supervised settings, DSE significantly outperforms BM25 on both webpage and slide retrieval. It achieves comparable performance to neural text retrievers like E5 on webpages and slightly trails Phi-3. On slide retrieval—which contains mixed text and visuals—DSE surpasses all text baselines. It also outperforms CLIP, confirming the benefit of the large vision-language model.

In zero-shot settings, DSE shows reasonable performance on TriviaQA and significantly outperforms all other methods on SlideVQA, demonstrating its ability to generalize without task-specific fine-tuning.

Increasing the number of crops (Cx, Cy) improves retrieval accuracy but reduces encoding speed. A moderate setting like 2x2 or 3x3 offers a good balance between effectiveness and efficiency.

Significance

DSE presents a clean, unified approach to multimodal document retrieval. By encoding screenshots directly, it eliminates the need for format-specific preprocessing and preserves all visual cues. This is particularly valuable for slide decks, scanned documents, or web pages where layout and images carry meaning. The use of Phi-3-vision with sub-image cropping allows fine-grained capture of text-heavy content. While the computational cost is higher than text-only methods, the trade-off is justified in scenarios where visual information is critical. DSE opens the door to simpler, more robust retrieval systems that can handle the messy reality of real-world documents.