Skip to content

Measuring What Truly Matters: Construct Validity of Large Language Model Benchmarks

Measuring what Matters: Construct Validity in Large Language Model Benchmarks

arXiv· arXiv:2511.04703· NeurIPS 2025· Andrew M. Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan, Chris Schmitz, Karolina Korgul, Hunar Batra, Oishi Deb, Emma Beharry, Cornelius Emde, Thomas Foster, Anna Gausen, María Grandury, Simeng Han, Valentin Hofmann, Lujain Ibrahim, Hazel Kim, Hannah Rose Kirk, Fangru Lin, Gabrielle Kaili-May Liu, Lennart Luettgau, Jabez Magomere, Jonathan Rystrøm, Anna Sotnikova, Yushi Yang, Yilun Zhao, Adel Bibi, Antoine Bosselut, Ronald Clark, Arman Cohan, Jakob Foerster, Yarin Gal, Scott A. Hale, Inioluwa Deborah Raji, Christopher Summerfield, Philip H. S. Torr, Cozmin Ududec, Luc Rocher, Adam Mahdi·Indexed by this site · 19 days ago· Originally publishedEvaluation
AI summary

A research team with 29 expert reviewers systematically reviewed 445 large language model benchmarks from top natural language processing and machine learning conferences, and found patterns in the phenomena under test, tasks, and scoring metrics that weaken conclusion validity. The research points out that reliably measuring abstract phenomena such as "safety" and "robustness" requires strong construct validity, that is, the measurement metrics genuinely represent the important parts of the phenomenon; such evaluations are critical for identifying model safety and robustness issues before deployment. To address these deficiencies, the paper proposes eight key recommendations and detailed actionable guidance for researchers and practitioners developing large language model benchmarks.

Read originalNeurIPS 2025