Fine-tuning LLM Agents without Fine-tuning LLMs

Analysis

Why This Paper Matters

This paper addresses a critical bottleneck in deploying LLM agents: the high cost and inflexibility of fine-tuning. Current approaches either rely on static, handcrafted reflection workflows that cannot adapt to new tasks, or require expensive gradient updates that are impractical for real-time learning. By introducing a memory-based online reinforcement learning paradigm that eliminates the need to fine-tune the underlying LLM, the authors open the door to truly continual, low-cost adaptation. This is particularly significant for deep research scenarios where agents must handle diverse, open-ended tasks without retraining.

The Memento agent demonstrates that a lightweight memory system can match or exceed the performance of training-based methods, while being far more efficient. The ability to add 4.7% to 9.6% absolute points on out-of-distribution tasks via case-based memory shows that the approach generalizes well, a key requirement for real-world deployment. This work shifts the focus from model-centric to memory-centric learning, which could democratize access to advanced AI agents by reducing computational requirements.

Technical Contributions

• Memory-augmented Markov Decision Process (M-MDP): Formalizes the agent-environment interaction with a neural case-selection policy that retrieves relevant past experiences from episodic memory to guide actions.
• Dual memory types: Supports both differentiable and non-parametric episodic memories, offering flexibility in implementation.
• Memory rewriting mechanism: Continuously updates stored experiences based on environmental feedback, enabling online learning without gradient updates.
• Efficient policy improvement: Achieved solely through memory reading (retrieval), avoiding costly parameter updates.
• Deep research instantiation: The Memento agent is specifically designed for complex, multi-step research tasks, demonstrating practical applicability.

Results

• GAIA benchmark: Top-1 on validation set with 87.88% Pass@3, and 79.40% on test set, outperforming all prior methods.
• DeepResearcher dataset: Achieves 66.6% F1 and 80.4% PM, surpassing state-of-the-art training-based methods.
• Out-of-distribution tasks: Case-based memory yields 4.7% to 9.6% absolute improvement, showing strong generalization.
• Efficiency: No gradient updates required, enabling low-cost continual adaptation in real-time settings.

Significance

This work represents a paradigm shift in LLM agent learning, moving from expensive model fine-tuning to efficient memory-based adaptation. It has broad implications for deploying AI agents in dynamic environments where tasks evolve continuously, such as scientific research, customer support, and autonomous systems. By decoupling learning from model parameter updates, the approach reduces computational costs and enables agents to learn from experience without forgetting previous knowledge. This could accelerate the development of generalist agents capable of open-ended skill acquisition, advancing the field toward more practical and scalable AI systems.