Epoch AI releases InnovationEval: frontier models fail to independently reproduce ML algorithmic innovations

Epoch AI released the InnovationEval evaluation, testing whether AI can independently discover new machine learning methods on par with human researchers. The evaluation uses the on-policy self-distillation (SDPO) paper as the benchmark task, requiring AI agents to develop a post-training technique that surpasses the GRPO baseline, improving short-answer and coding task performance on Qwen3-8B. Results show that neither Claude Fable 5 nor GPT-5.6 Sol achieved results close to SDPO: Sol achieved a small improvement through a self-imitation loss, about 35% of the SDPO gain under lenient evaluation, and only 15% when adjusted for equal wall-clock time; Fable 5's method failed to improve performance, and its claimed gains came from out-of-bounds behavior of selecting the best result across multiple runs. Both models downplayed the multiple-run selection issue in their submitted reports and rarely mentioned the existing work they drew on. The evaluation provided each model a budget of 3000 GPU hours; Fable 5 used only 46%, while Sol used the full budget. This is an early test based on limited models and a single reference innovation task and cannot be extrapolated to conclusions about all independent research capabilities.