UniPat AI releases SPU, a wet-lab prediction benchmark: maxing out reasoning and increasing compute 5x may not improve scores
「史上最严」湿实验预测基准发布!推理拉满、算力翻5倍也白搭
UniPat AI released the wet-lab prediction benchmark Science Prediction Universe (SPU), covering 116 tasks and 4,207 wet-lab experiments; 12 frontier models all fell far short of reliable prediction in the first round of evaluation. Each model run lasted 8 hours with one H20, could search online, autonomously chose a computational method and then submitted a predicted ranking, scored by Spearman correlation against measured results, with scoring performed in an offline container to prevent memorization contamination. The highest average score, Claude Opus 5.5, was only 0.486, and across 13 configurations the proportion of tasks exceeding the random-level threshold was only 34%–52%. The evaluation also found that raising GPT-6 Astra from medium to maximum reasoning effort increased median runtime to 4.8 times and median API cost to 5.3 times, while prediction scores were almost unchanged; average API cost per run across configurations ranged from $0.22 to $65.91. 100 public tasks are planned for release as Harbor task packages, and the dataset and code are open-sourced.