Multiagent Finetuning

Analysis

Why This Paper Matters

Large language models (LLMs) have shown remarkable capabilities, but their self-improvement often plateaus due to homogenization of training data. Traditional methods like iterative self-training (e.g., STaR) fine-tune a single model on its own correct outputs, which gradually reduces response diversity and leads to overfitting. This paper introduces Multiagent Finetuning, a novel approach that leverages a society of specialized agents—generation and critic models—trained on data generated through multiagent debate. By finetuning each agent on its own unique set of correct responses and debate sequences, the method preserves diversity and enables iterative improvement over multiple rounds. This is significant because it offers a scalable, label-free path to enhance reasoning without sacrificing the variety of reasoning chains.

How It Works

The method consists of two main components: multiagent debate to generate training data, and multiagent finetuning to specialize each model.

Multiagent Debate

A set of N LLM agents (initially identical) each generate an initial response to a problem. Then, over M rounds, each agent sees its own prior response and a summary of all other agents' responses, and produces a new response. The final answer is determined by majority vote among the last-round outputs. This process yields high-quality answers but often produces stylistically similar outputs across agents.

Finetuning Generation and Critic Agents

To break the homogeneity, the authors propose finetuning two types of agents:

• Generation agents are finetuned on their own initial responses that match the final debate answer. Each generation agent gets a unique dataset of (input, correct initial response) pairs, promoting specialization and diverse reasoning.

• Critic agents are finetuned on full debate sequences. Crucially, they include sequences where the initial answer was wrong but the final answer was correct (DC- dataset), teaching the critic to correct errors. They also include sequences where the answer was correct throughout (DC+ dataset). This equips critics to both maintain correct answers and fix mistakes.

After finetuning, the updated generation and critic agents are used in the next iteration of debate to generate new training data, enabling iterative self-improvement.

Inference

At inference, generation agents produce initial responses, then critic agents refine them over multiple debate rounds. Each agent summarizes others' responses to remove redundancy while preserving key details. Final answer is via majority vote.

Results

The proposed multiagent finetuning outperforms all baselines across datasets and model sizes. Notably, it beats the iterative self-training baseline (STaR) even with a single finetuning iteration, despite STaR using ground-truth labels. Multiple iterations further improve performance, while single-agent finetuning saturates and declines.

Crucially, diversity of responses is maintained or even increased across iterations, whereas single-agent finetuning reduces diversity.

Significance

Multiagent Finetuning offers a practical, label-free method for LLM self-improvement that avoids the diversity collapse common in single-agent approaches. By training a society of specialized agents, it not only boosts accuracy but also preserves the variety of reasoning chains, which is valuable for robustness and interpretability. This work opens the door to more sophisticated multiagent training paradigms where diversity is explicitly encouraged, potentially leading to more capable and adaptable AI systems.