Skip to content

Visual Cipher Bench: evaluating long-horizon visual reasoning in LLM agents

Visual Cipher Bench: Benchmarking LLM Agents on Long-Horizon Visual Reasoning

arXiv· arXiv:2611.16861· Deyao Hong, Jiayi Jiang, Yuchun Chen, Yixuan Shi, Wenyi Li, Zheng Jiang, Chengxing Xie, Mingyu Su, Qingle Liu, Mingju Gao, Yizhe Chi, Youjie Zheng, Calvin Xiao, Kaisen Yang, Huan-ang Gao·Indexed by this site · 1 days ago· Originally publishedAI updates
AI summary

Researchers introduce Visual Cipher Bench, which uses 40 original visual puzzles to evaluate long-horizon visual reasoning in LLM agents; the puzzles cover four categories: spatial manipulation and simulation, logical constraints and planning, rule induction, and knowledge-based decoding, each providing only one to three black-and-white line drawings and brief submission instructions, and the agent must discover the solving mechanism entirely from the images. Across 29 reasoning effort configurations of 8 frontier models, the strongest, GPT-6 Astra (max), achieved only 26.7% accuracy, and 5 of the 8 models were below 1%. On a subset of 10 tasks, a non-spoiler mechanism hint raised accuracy from 12.2% to 51.1%. The authors therefore conclude that the main bottleneck of current agents is not executing multi-step plans, but autonomously discovering rules from visual evidence.

Related events and news

1 events · 1 news mentions

News that mentions this paper

Einsia AI发布长程视觉推理基准 最强模型仅26.7%

Read originalarxiv.org