Einsia AI releases a long-horizon visual reasoning benchmark; the strongest model reaches only 26.7%
Start here
On November 12, researchers released Visual Cipher Bench on arXiv, using 40 original visual puzzles to evaluate the long-horizon visual reasoning of LLM agents. The puzzles cover four categories: spatial manipulation and simulation, logical constraints and planning, rule induction, and knowledge decoding. Agents can only discover the solving mechanism from one to three black-and-white line drawings. Across 29 reasoning effort configurations of 8 frontier models, the strongest, GPT-6 Astra (max), achieved only 26.7% accuracy, and 5 of the 8 models scored below 1%. On a subset of 10 questions, after adding one sentence to each of the three GPT-6 models giving a core rule hint without spoilers, accuracy rose from 12.2% to 51.1%. The authors therefore argue that the main bottleneck of current agents is not executing multi-step plans, but autonomously discovering rules from visual evidence. On November 17, Machine Heart reported that the benchmark was released by Navers Lab under Einsia AI, with 8 models making 3480 attempts in total, and none of the 29 configurations exceeding 30%; DeepSeek V4.1 Flash, Qwen 3.8 Max, and Kimi K3.1 Pro did not solve a single question correctly, and the rule induction category had only 5 correct out of 870 attempts.
Generated by AI from reports · Updated 1 days ago
Progress timeline
- WebSynced (Jiqizhixin)· Media coverageThe strongest, GPT-6 Astra, scores only 26.7%, as Visual Cipher Bench released by Einsia AI stumps top AI(opens the original post in a new tab)
- WeChat · Synced (Jiqizhixin)Einsia AI releases Visual Cipher Bench: the strongest, GPT-6 Astra, answers only 26.7% correctly
Einsia AI's Navers Lab released Visual Cipher Bench, an agent visual reasoning benchmark. Across 3480 attempts by 8 frontier models, the strongest configuration, GPT-6 Astra (max), answered only 26.7% correctly. The benchmark contains 40 original visual puzzles and 67 black-and-white line drawings, with clues drawn only in the images, divided into four categories: spatial manipulation and simulation, logical constraints and planning, rule induction, and knowledge decoding. None of the 29 reasoning-intensity configurations exceeded 30%; DeepSeek V4.1 Flash, Qwen 3.8 Max and Kimi K3.1 Pro each got not a single question right; in the rule induction category, only 5 of 870 attempts were correct. The research team additionally took 10 questions and, after adding one core-rule hint for each of three GPT-6 models, overall accuracy rose from 12.2% to 51.1%.
- arXivVisual Cipher Bench: evaluating long-horizon visual reasoning in LLM agents
Researchers introduce Visual Cipher Bench, which uses 40 original visual puzzles to evaluate long-horizon visual reasoning in LLM agents; the puzzles cover four categories: spatial manipulation and simulation, logical constraints and planning, rule induction, and knowledge-based decoding, each providing only one to three black-and-white line drawings and brief submission instructions, and the agent must discover the solving mechanism entirely from the images. Across 29 reasoning effort configurations of 8 frontier models, the strongest, GPT-6 Astra (max), achieved only 26.7% accuracy, and 5 of the 8 models were below 1%. On a subset of 10 tasks, a non-spoiler mechanism hint raised accuracy from 12.2% to 51.1%. The authors therefore conclude that the main bottleneck of current agents is not executing multi-step plans, but autonomously discovering rules from visual evidence.
Related discussion
- X@AiwithZohaibGPT-6 Astra gets to 26.7% on Visual Cipher Bench at max effort, up from 5.0% at low, and that's the best anyone did, so…(opens the original post in a new tab)
67 likes·Nov 18 23:41Collect
- X@RAVIKUMARSAHU78Visual Cipher Bench cranked GPT-6 Astra's effort up and it climbed from 5.0% to 26.7%, but Claude Opus 5.5 slid from 21…(opens the original post in a new tab)
167 likes·Nov 18 23:41Collect
- X@Mr_Shahbaz_AiGPT-6 Astra on max effort gets 26.7% on Visual Cipher Bench and that's the best of the lot. Three other models didn't s…(opens the original post in a new tab)
97 likes·Nov 18 23:41Collect
- X@EinsiaAICoding agents can now grind through long-horizon tasks, as long as they get a written spec and a test suite. What happ…(opens the original post in a new tab)
124 likes·Nov 18 23:41Collect
- WebSynced (Jiqizhixin)The strongest, GPT-6 Astra, scores only 26.7%, as Visual Cipher Bench released by Einsia AI stumps top AI(opens the original post in a new tab)
Attention trend
New sources per day
- Nov 17: 3 new sources (1 outlets, 2 accounts)
- Nov 18: 2 new sources (0 outlets, 2 accounts)
- Nov 19: 0 new sources (0 outlets, 0 accounts)
- Nov 20: 0 new sources (0 outlets, 0 accounts)