Robustness tests in the Benchmark Health Index BHI paper show that after removing 30% of models, the Spearman correlation of benchmark rankings remains above 0.97
Researchers from SKYEVAL and Alibaba DAMO Academy propose the Benchmark Health Index BHI for auditing the benchmarks used to compare LLMs; the extended arXiv paper (arXiv:2602.11674v2) analyzes 158 benchmarks reported by 117 models. BHI combines three signals, namely a benchmark's ability to distinguish different models by score, the remaining challenge space before saturation, and the breadth of adoption of the benchmark by capable models; model capability is first estimated via leave-one-benchmark-out calibration without relying on the benchmark under audit, and then a total index is synthesized using CRITIC weighting. Robustness tests in the paper show that after removing 30% of models, the Spearman correlation of benchmark rankings remains above 0.97. The paper also notes that robustness to missing records does not mean the reports are fully comparable, as prompts, tools and evaluation conditions may differ, and that short-term saturation predictions cannot determine the exact date a benchmark will fail; the per-dimension metrics still have reference value beyond the composite ranking.
Removing 30% of models still leaves benchmark rankings with a Spearman correlation above 0.97 in the BHI paper's robustness tests.
Researchers from SKYEVAL and Alibaba's DAMO Academy propose the Benchmark Health Index to audit the benchmarks used to compare language models. The expanded arXiv version analyzes 158 benchmarks drawn from reports for 117 models.
BHI combines three signals: how well scores separate models, how much challenge remains before saturation, and how broadly capable models adopt a benchmark. Leave-one-benchmark-out calibration estimates model capability without relying on the benchmark currently under inspection. CRITIC weighting combines the complementary dimensions into an overall index.
The result is a tool for selecting and revisiting evaluation sets as models improve. Robustness to missing records does not make the underlying reports perfectly comparable: prompting, tools and evaluation conditions can differ. Nor can a short-term saturation projection determine a benchmark's exact expiry date. The individual dimensions remain useful alongside the composite ranking.
Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs
Paper: https://arxiv.org/abs/2602.11674v2
Our Report: https://www.jiqizhixin.com/articles/2026-10-31-4
📬 #PapersAccepted by Jiqizhixin