AlignBench: evaluating the Chinese alignment ability of large language models

The researchers release AlignBench, a multi-dimensional benchmark for evaluating the alignment ability of Chinese large language models, containing 8 main categories, 683 queries derived from real scenarios, and manually verified reference answers. Knowledge-intensive queries also come with URLs and citations collected by annotators from reliable web sources as evidence. Automatic evaluation uses a rule-calibrated multi-dimensional LLM-as-Judge method, combined with Chain-of-Thought to generate scoring explanations; the evaluation code, data and model generation results are open-sourced on GitHub. The paper states that since its release, the benchmark has been used by Chinese models such as ChatGLM, Qwen, DeepSeek, Yi, Baichuan and Abab to evaluate their Chinese alignment ability.