AlpacaEval
FreeAn Automatic Evaluator for Instruction-following Language Models using Nous benchmark suite.
About AlpacaEval
AlpacaEval is an LLM-based automatic evaluation system for instruction-following language models, designed to be fast, cheap, and reliable. It leverages the AlpacaFarm evaluation set, which consists of diverse but generally simple instructions, and uses GPT-4-based auto-annotators to compare model responses against reference responses (Davinci003 for AlpacaEval 1.0, GPT-4 Preview for AlpacaEval 2.0). The system calculates win rates and supports length-controlled (LC) win rates to mitigate length bias in annotations. AlpacaEval demonstrates high agreement with human annotations, and its leaderboard rankings correlate strongly with human-based evaluations. The tool is open-source, welcomes community contributions for new models, evaluators, and evaluation sets, and includes a Discord for support. Limitations include a focus on simple instructions, lack of safety evaluation, and residual length biases.
Key Features
Pros & Cons
- Fast, cheap, and reliable automatic evaluation
- High correlation with human annotations, providing trustworthy rankings
- Open-source and actively maintained with community contributions
- Length-controlled win rates mitigate length bias in evaluations
- Does not evaluate model safety
- Evaluation set consists mainly of simple instructions, limiting generalizability
- Auto-annotator win rates still correlated with length despite LC mitigation
- Not a comprehensive or gold-standard evaluation; should be used alongside other methods