Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs

Free

Improving Chain-of-Thought Reasoning in LLMs via Preference Optimization

FreeFree tier

Inputs: textOutputs: text

Type

Open Source

Visit Website GitHub (40 stars)

About Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs

Chain of Preference Optimization (CPO) is a fine-tuning method for large language models (LLMs) that improves chain-of-thought (CoT) reasoning by aligning each reasoning step with the more deliberate paths discovered by tree-of-thought (ToT) search. Unlike ToT, which incurs significant inference overhead due to tree searching, CPO transfers the benefits of exploration into the model weights via preference optimization, enabling CoT decoding to achieve comparable or better performance without extra inference cost. The method is evaluated on complex problem-solving tasks including question answering, fact verification, and arithmetic reasoning, and is shown to significantly boost LLM performance. CPO was accepted at NeurIPS 2024 and the code is publicly available.

Key Features

Fine-tunes LLMs using preference information from tree-of-thought (ToT) search trees

Aligns each chain-of-thought reasoning step with ToT-identified optimal paths

Eliminates the inference complexity overhead of tree-of-thought decoding

Significantly boosts performance on question answering, fact verification, and arithmetic reasoning

Open-source implementation available (NeurIPS 2024)

Pros & Cons

Pros

Reduces inference burden compared to tree-of-thought decoding while achieving similar or better results
Enhances performance on a variety of complex reasoning tasks
Open-source code enables reproducibility and further research
Accepted at top conference (NeurIPS 2024), indicating quality
Method is model-agnostic and can be applied to different LLM architectures

Cons

Requires fine-tuning on task-specific data, not a zero-shot solution
Training relies on constructing tree-of-thought search trees, which can be computationally expensive
Effectiveness may depend on the quality of the tree-of-thought exploration during training
Limited evaluation compared to broader real-world deployment scenarios

Best For

Improving complex reasoning capabilities in large language modelsQuestion answering tasks requiring multi-step reasoningFact verification with logical chain-of-thoughtArithmetic reasoning and mathematical problem solvingAny application where tree-of-thought would be beneficial but inference cost is a concern

FAQ

What is Chain of Preference Optimization (CPO)?

CPO is a fine-tuning method that improves chain-of-thought reasoning in LLMs by aligning each reasoning step with the preferences derived from tree-of-thought search trees, allowing CoT to match ToT performance without the inference overhead.

How does CPO differ from chain-of-thought (CoT) and tree-of-thought (ToT)?

CoT generates a single reasoning path, which may be suboptimal. ToT explores multiple paths via tree search, which is expensive. CPO fine-tunes the model so that CoT decoding implicitly follows the better paths discovered by ToT, reducing inference cost.

Is the CPO code available?

Yes, the paper states that code is available at an anonymous GitHub repository, and the method is open-source.