Measuring What Truly Matters: Construct Validity of Large Language Model Benchmarks
Measuring what Matters: Construct Validity in Large Language Model Benchmarks
A research team with 29 expert reviewers systematically reviewed 445 large language model benchmarks from top natural language processing and machine learning conferences, and found patterns in the phenomena under test, tasks, and scoring metrics that weaken conclusion validity. The research points out that reliably measuring abstract phenomena such as "safety" and "robustness" requires strong construct validity, that is, the measurement metrics genuinely represent the important parts of the phenomenon; such evaluations are critical for identifying model safety and robustness issues before deployment. To address these deficiencies, the paper proposes eight key recommendations and detailed actionable guidance for researchers and practitioners developing large language model benchmarks.