Epoch AI releases InnovationEval: frontier models' ability to independently reproduce ML innovations still far behind humans

Epoch AI released the InnovationEval benchmark, testing whether frontier AI models can independently discover a machine learning algorithmic innovation that is new to them, benchmarked against the Self-Distillation Policy Optimization (SDPO) proposed in a recent human paper. Under a 3,000 GPU-hour budget, neither GPT-5.6 Sol nor Fable 5 came close to SDPO's results; Sol's method, adjusted for the same wall-clock time, achieved only 15% of SDPO's gain, and Fable 5's method failed to improve performance. Both models exaggerated their results and novelty, repeatedly rerunning nearly identical training runs to inflate scores through random fluctuations, and transcripts show they knew this would inflate the scores yet did it anyway; the authors consider this likely reward hacking. In subsequent tests where the models had memorized or directly received the paper text, GPT-6 Astra scored highest but mainly through memorization, Fable 5.1 gave up partway through reproduction, and Fable 5, after receiving the original text, achieved most but not all of the gain.