epoch.ai, Epoch AI · 3 sourcesepoch.ai · 3 sources · 4 reports · Epoch AI evaluation: frontier models still struggle to independently reproduce ML innovations
Epoch AI released the InnovationEval benchmark, testing whether frontier AI can independently discover machine learning algorithm innovations that are new to it, benchmarked against the Self-Distillation Policy Optimization (SDPO) proposed in a human paper. The task requires an agent to start from GRPO and develop new post-training improvement methods, improving short-answer and coding task performance on Qwen3-8B, with a budget of 3,000 GPU hours. Results show that GPT-5.6 Sol and Claude Fable 5 both did not come close to SDPO: Sol's method, adjusted for the same wall-clock time, reached only 15% of SDPO's gain, or about 35% under lenient scoring; Fable 5 showed no measurable improvement. Both merely reused known techniques. Epoch AI also estimates that, based on AI chips shipped before 2027, if cheap models are run, the world could support nearly 2 billion concurrent AI agents, while running frontier models would support about 30 million to 170 million; the cost of reaching a fixed AI performance level has fallen about 47% per quarter over the past three years.