Neura Market
ACLUE logo

ACLUE

Free

an evaluation benchmark focused on ancient Chinese language comprehension.

FreeFree tier
Inputs: text
Type
Open Source
Visit Website GitHub (38 stars)

About ACLUE

ACLUE (Ancient Chinese Language Understanding Evaluation) is a benchmark designed to evaluate the performance of large language models (LLMs) on ancient Chinese language comprehension. It consists of 15 multiple-choice tasks spanning five domains: vocabulary (e.g., polysemy, interchangeable characters, named entity recognition), syntax (sentence segmentation), semantics (couplet completion, poetry line prediction), reasoning (poetry quality assessment, reading comprehension, poetry appreciation, emotion classification), and knowledge (ancient Chinese knowledge, traditional culture, medical ancient texts, literary history, historical phonetics). The dataset covers texts from the Xia Dynasty (c. 2070 BCE) to the Ming Dynasty (1368 CE). A leaderboard tracks zero-shot performance for models like ChatGLM2-6B, ChatGPT, BLOOMZ-7B, and others. The project is open source under the MIT license, with data licensed under CC BY-NC-SA 4.0. A paper describing ACLUE was presented at the Ancient Language Processing Workshop (ALP 2023).

Key Features

15 tasks covering vocabulary, syntax, semantics, reasoning, and knowledge
Multiple-choice format with four options and one correct answer
Zero-shot evaluation leaderboard for various LLMs
Data spans historical periods from Xia to Ming dynasties
Open source under MIT license (code) and CC BY-NC-SA 4.0 (data)
Accompanied by a published academic paper (ALP 2023)

Pros & Cons

Pros
  • Comprehensive coverage of multiple aspects of ancient Chinese language
  • Structured evaluation with a public zero-shot leaderboard
  • Open source and freely available for research
  • Based on a peer-reviewed academic paper
  • Includes diverse task types from vocabulary to reasoning
Cons
  • Limited to ancient Chinese language only, not multilingual
  • Multiple-choice format may not capture nuanced generative understanding
  • Relatively small number of tasks (15) compared to broader benchmarks
  • Leaderboard currently shows low performance even for top models, indicating difficulty

Best For

Evaluating large language models' comprehension of ancient ChineseAcademic research on ancient language processing and NLPTraining and improving models for classical Chinese tasksBenchmarking progress in historical language understanding

FAQ

What tasks are included in ACLUE?
ACLUE includes 15 tasks divided into five categories: vocabulary (3 tasks), syntax (1 task), semantics (2 tasks), reasoning (4 tasks), and knowledge (5 tasks). Examples include classical Chinese polysemy, sentence segmentation, couplet completion, reading comprehension, and traditional culture knowledge.
What is the data format?
Each question is a multiple-choice item with four options (A, B, C, D) and one correct answer. Data is stored in CSV files with columns for the question, four options, and the correct answer letter.
How can I contribute my model's results?
The project encourages researchers to contribute zero-shot evaluation results by contacting the authors or submitting a pull request to the GitHub repository.
What license does ACLUE use?
The code is released under the MIT License, and the dataset is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).
Is there a paper associated with ACLUE?
Yes, a paper titled 'Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE' by Yixuan Zhang and Haonan Li was presented at the Ancient Language Processing Workshop (ALP 2023).