GitHub
FreeImproving Factuality and Reasoning in Language Models through Multiagent Debate
FreeFree tier
Inputs: textOutputs: text
About GitHub
This repository provides a preliminary implementation of the ICML 2024 paper 'Improving Factuality and Reasoning in Language Models through Multiagent Debate.' It includes code for running multiagent debate experiments on arithmetic, grade school math (GSM), biography generation, and MMLU tasks. The multiagent debate approach generates multiple responses from language models and debates them to improve factual accuracy and reasoning. The code is written in Python and open source under the composable-models organization.
Key Features
Multiagent debate framework to improve LLM factuality and reasoning
Code for arithmetic, grade school math (GSM), biography, and MMLU tasks
Generation and evaluation scripts for each benchmark
Preliminary implementation with more tasks to be released
Compatible with open-source LLMs via community implementations
Pros & Cons
Pros
- Based on peer-reviewed research accepted at ICML 2024
- Directly addresses factuality and reasoning, key LLM challenges
- Supports multiple established benchmarks (Math, GSM, MMLU, Biography)
- Open source and freely available for modification and reuse
Cons
- Preliminary implementation; may not cover all tasks or be production-ready
- Requires multiple LLM calls per query, increasing computational cost
- No official releases or packaged installation; usage requires manual setup
Best For
Improving factual accuracy of language model outputsEnhancing reasoning capabilities in LLMs through debateAcademic research on multiagent systems and LLM alignmentBenchmarking factuality on math, GSM, biography, MMLU datasets