A URL Fetch Gemini Processor to be used with Gemini's genai-processors

genai-processors-url-fetch

A URL Fetch Processor for Google's genai-processors framework that detects URLs in text, fetches their content concurrently, and yields new ProcessorParts containing the page content.

UrlFetchProcessor

The UrlFetchProcessor is a PartProcessor that detects URLs in incoming text parts, fetches their content concurrently, and yields new ProcessorParts containing the page content. It is a powerful and secure tool for enabling AI agents to access and process information from the web.

Motivation

Many advanced AI applications, especially those involving Retrieval-Augmented Generation (RAG) or agentic behavior, need to interact with the outside world. This processor provides the fundamental capability of "reading" a webpage.

• Enables RAG: Fetches the content of source URLs so an LLM can use up-to-date information to answer questions.
• Automates Research: Allows an agent to follow links to gather context for a research task.
• Simplifies Tooling: Abstracts away the complexities of asynchronous HTTP requests, rate-limiting, security validation, and HTML parsing.

Installation

Install the package using pip:

pip install genai-processors-url-fetch

For enhanced content processing with markitdown support:

pip install genai-processors-url-fetch[markitdown]

Or using uv (recommended):

uv add genai-processors-url-fet