OmniParser

Analysis

Why This Paper Matters

Vision-language models (VLMs) like GPT-4V have shown remarkable ability to understand images, but they often fail at the fine-grained task of interacting with graphical user interfaces (GUIs). The core problem is that VLMs lack a structured understanding of screen elements—they cannot reliably tell which icons are clickable, what each icon does, or where to click based on a natural language instruction. This gap limits the development of autonomous GUI agents that could automate tasks like form filling, web navigation, or app control. OmniParser directly addresses this by providing a robust screen parsing pipeline that converts raw screenshots into a structured, DOM-like representation, enabling VLMs to generate grounded actions with high accuracy.

How It Works

OmniParser breaks down the GUI interaction task into two steps: understanding the current screen (content and icon functions) and predicting the next action. To achieve this, it integrates three components: a fine-tuned interactable icon detection model, a fine-tuned icon description model, and an OCR module. The outputs are combined into a structured representation that includes bounding boxes with unique IDs and local semantics (text and icon descriptions).

Interactable Region Detection: Instead of asking GPT-4V to predict raw coordinates, OmniParser uses a Set-of-Marks approach: it overlays bounding boxes of interactable icons on the screenshot and asks the model to output the bounding box ID. A detection model is fine-tuned on a dataset of 67k screenshots with bounding boxes derived from DOM trees of 100k popular URLs. An OCR module extracts text bounding boxes, and overlapping boxes (over 90% IoU) are merged. Each box gets a unique numeric ID placed to minimize overlap.

Local Semantics: For each detected icon, a fine-tuned BLIP-v2 model generates a description of its functionality (trained on 7k GPT-4o-generated icon-description pairs). For text boxes, the OCR text is used. This local semantics is crucial: without it, GPT-4V often hallucinates or fails to link task descriptions to correct icon IDs.

Results

On the SeeAssign benchmark (112 tasks), adding local semantics improved GPT-4V's accuracy from 0.705 to 0.938. On ScreenSpot, OmniParser outperformed GPT-4V baselines and specialized models like SeeClick and CogAgent across mobile, desktop, and web platforms. The fine-tuned detection model added 4.3% accuracy over raw Grounding DINO.

On Mind2Web, OmniParser + GPT-4V surpassed all screenshot-only models, with gains of +4.1% and +5.2% in Cross-Website and Cross-Domain categories over GPT-4V+SOM. On AITW, it achieved a 4.7% overall improvement over the best GPT-4V + history baseline.

Significance

OmniParser V2 further improves with 60% faster inference and state-of-the-art 39.6% accuracy on the challenging ScreenSpot Pro benchmark (GPT-4o scored 0.8%). The release of OmniTool—a dockerized Windows system supporting multiple LLMs (OpenAI, DeepSeek, Qwen, Anthropic)—makes this approach practical for real-world agent development. By turning any LLM into a computer-use agent, OmniParser bridges the gap between vision-language understanding and actionable GUI automation, with broad implications for accessibility, productivity, and AI-driven task completion.