Skip to content

DominoBench: evaluating multi-step physical reasoning from static images

DominoBench: Evaluating Multi-Step Physical Reasoning from Still Images

Hugging Face Daily Papers· arXiv:2611.20385· Emilio Ferrara·Indexed by this site · 2 days ago· Originally publishedAI updates· Its place in the scan record →
AI summary

Researchers release the DominoBench benchmark; tests show that current vision-language and multimodal models cannot reliably predict physical outcomes from static images without assistance. The benchmark uses a physics simulator to judge outcomes from static images of 2D rigid-body scenes, contains 854 entries and 427 minimal pairs, in each of which a single quantity changing by one to three cells determines the life or death of one person, and each entry comes with a four-level hint ladder ending in the exact calculation; 30 models from 10 model families were tested. Frontier models from Anthropic, Google and OpenAI, without assistance, judged the person whose fate flips correctly at most 72% of the time, and answered both questions in a pair correctly at most 34% of the time; pointing out the decisive step or providing exact geometry as text helped little. Once the decisive calculation and the required quantities (including velocities that static images cannot show) were written out, the same models reached 82-92% on this item, while small open-weight models remained near chance.

Read originalhuggingface.co