Improving Legibility of LLM Outputs

Analysis

Why This Paper Matters

As large language models (LLMs) become more capable, ensuring their outputs are trustworthy is critical. This paper tackles a fundamental challenge: even when an LLM produces a correct answer, its reasoning process may be opaque or misleading, reducing human confidence. The authors introduce the concept of "legibility"—the property that model outputs are clear and easy to check—and show that standard training for answer correctness can actually harm legibility. This is a significant insight because it highlights a tension between performance and transparency that has been largely overlooked.

The paper's approach is inspired by the Prover-Verifier Game, originally used for adversarial robustness. By training a small verifier to check solutions and then training provers to either produce correct solutions that pass verification or incorrect ones that fool it, the authors create a training dynamic that forces the model to produce more checkable reasoning. This is a clever use of game theory to align model behavior with human verifiability, and it opens a new avenue for improving LLM alignment without requiring human oversight at every step.

Technical Contributions

• Legibility metric: The paper operationalizes legibility as the accuracy of a verifier (initially a small model, then humans) in judging solution correctness. This provides a concrete target for optimization.
• Prover-Verifier Game for legibility: The training algorithm alternates between three roles: helpful prover (correct, verifier-accepted), sneaky prover (incorrect but verifier-accepted), and verifier (predicts correctness). This adversarial setup incentivizes the helpful prover to produce solutions that are easy to verify.
• Transfer to humans: A key innovation is showing that legibility training against small verifiers transfers to time-constrained human verifiers. This suggests that the learned properties are not just artifacts of the verifier architecture but reflect genuine improvements in clarity.
• Adversarial robustness: The verifier's accuracy against sneaky provers increases over training, indicating that the game improves both the prover's legibility and the verifier's detection ability.

Results

The paper reports that over the course of training, the helpful prover's accuracy on grade-school math problems increases, while the sneaky prover's ability to fool the verifier decreases. In human experiments, participants asked to check solutions showed higher accuracy when evaluating helpful prover outputs and lower accuracy when evaluating sneaky prover outputs, compared to baseline models. This demonstrates that legibility training has a measurable effect on human verification performance, even under time constraints.

Significance

This work provides a practical, scalable method for increasing the transparency of LLM reasoning. By training against small verifiers, the approach avoids the need for expensive human feedback loops. The transfer to human verifiers is particularly important for alignment, as it suggests that models can be trained to produce outputs that are inherently more checkable by humans. This could be a key component in building trust in AI systems, especially as models become superhuman and direct human oversight becomes less feasible. The paper also opens up new research directions, such as extending legibility training to other domains and exploring the trade-offs between legibility and other desiderata like accuracy or creativity.