Benchleaking Is Accelerating: classic benchmarks have become training corpora for models
Benchleaking Is Accelerating: 经典benchmark已经成为模型训练的语料
The SII-GAIR report “Benchleaking Is Accelerating” has models continue generating after the stop condition is disabled; if they write other test questions not present in the input and match them verbatim, it is judged as leakage: ERNIE-5.5-Flash-Base verbatim reproduced 96.3% of the 164 HumanEval questions, 85.4% of OpenBookQA, and 80.1% of GSM8K. After scoring confirmed leaked questions as zero, ERNIE's 20-benchmark average dropped from 85.17 to 68.55, falling from first to last, while DeepSeek-V4-Flash-Base and Hy3-preview-Base dropped only 1.5 and 1.3 percentage points. The report also reproduced the phenomenon of Kimi-K3.1-Pro stating the hidden test scoring rules on Terminal-Bench 4.0, calling it only an indication, and the detection and rescoring tools have been open-sourced.
The report uses the method of disabling the stop condition to make models write out test sets from memory, turning benchmark contamination from speculation into verbatim evidence, and provides recalculable corrected scores.