Visual Cipher Bench: evaluating long-horizon visual reasoning in LLM agents
Visual Cipher Bench: Benchmarking LLM Agents on Long-Horizon Visual Reasoning
Researchers introduce Visual Cipher Bench, which uses 40 original visual puzzles to evaluate long-horizon visual reasoning in LLM agents; the puzzles cover four categories: spatial manipulation and simulation, logical constraints and planning, rule induction, and knowledge-based decoding, each providing only one to three black-and-white line drawings and brief submission instructions, and the agent must discover the solving mechanism entirely from the images. Across 29 reasoning effort configurations of 8 frontier models, the strongest, GPT-6 Astra (max), achieved only 26.7% accuracy, and 5 of the 8 models were below 1%. On a subset of 10 tasks, a non-spoiler mechanism hint raised accuracy from 12.2% to 51.1%. The authors therefore conclude that the main bottleneck of current agents is not executing multi-step plans, but autonomously discovering rules from visual evidence.
Related events and news
1 events · 1 news mentions
News that mentions this paper
Einsia AI发布长程视觉推理基准 最强模型仅26.7%
- 公众号 机器之心cites the paper's id or linkEinsia AI发布Visual Cipher Bench:最强GPT-6 Astra仅答对26.7%
论文题目:Visual Cipher Bench: Benchmarking LLM Agents on Long-Horizon Visual Reasoning