基准健康指数BHI论文的鲁棒性测试显示,移除30%模型后基准排名Spearman相关性仍高于0.97
SKYEVAL 与阿里巴巴达摩院的研究者提出基准健康指数 BHI,用于审计比较 LLM 所用的基准,扩展版 arXiv 论文(arXiv:2602.11674v2)分析了来自 117 个模型报告的 158 个基准。BHI 综合三个信号,即分数区分不同模型的能力、饱和前剩余的挑战空间,以及能力强模型采用该基准的广度;先通过留一基准(leave-one-benchmark-out)校准在不依赖被审基准的情况下估计模型能力,再用 CRITIC 加权合成总指数。论文的鲁棒性测试显示,移除 30% 的模型后,基准排名的 Spearman 相关性仍高于 0.97。论文同时指出,对缺失记录的鲁棒性并不意味着各报告完全可比,提示词、工具和评测条件可能存在差异,短期饱和预测也无法确定基准的确切失效日期,各维度指标在综合排名之外仍有参考价值。
Removing 30% of models still leaves benchmark rankings with a Spearman correlation above 0.97 in the BHI paper's robustness tests.
Researchers from SKYEVAL and Alibaba's DAMO Academy propose the Benchmark Health Index to audit the benchmarks used to compare language models. The expanded arXiv version analyzes 158 benchmarks drawn from reports for 117 models.
BHI combines three signals: how well scores separate models, how much challenge remains before saturation, and how broadly capable models adopt a benchmark. Leave-one-benchmark-out calibration estimates model capability without relying on the benchmark currently under inspection. CRITIC weighting combines the complementary dimensions into an overall index.
The result is a tool for selecting and revisiting evaluation sets as models improve. Robustness to missing records does not make the underlying reports perfectly comparable: prompting, tools and evaluation conditions can differ. Nor can a short-term saturation projection determine a benchmark's exact expiry date. The individual dimensions remain useful alongside the composite ranking.
Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs
Paper: https://arxiv.org/abs/2602.11674v2
Our Report: https://www.jiqizhixin.com/articles/2026-10-31-4
📬 #PapersAccepted by Jiqizhixin