Skip to content
EventMonitoring

Einsia AI releases a long-horizon visual reasoning benchmark; the strongest model reaches only 26.7%

2 reports2 reporting sourcesLatest report

Start here

AI overview

On November 12, researchers released Visual Cipher Bench on arXiv, using 40 original visual puzzles to evaluate the long-horizon visual reasoning of LLM agents. The puzzles cover four categories: spatial manipulation and simulation, logical constraints and planning, rule induction, and knowledge decoding. Agents can only discover the solving mechanism from one to three black-and-white line drawings. Across 29 reasoning effort configurations of 8 frontier models, the strongest, GPT-6 Astra (max), achieved only 26.7% accuracy, and 5 of the 8 models scored below 1%. On a subset of 10 questions, after adding one sentence to each of the three GPT-6 models giving a core rule hint without spoilers, accuracy rose from 12.2% to 51.1%. The authors therefore argue that the main bottleneck of current agents is not executing multi-step plans, but autonomously discovering rules from visual evidence. On November 17, Machine Heart reported that the benchmark was released by Navers Lab under Einsia AI, with 8 models making 3480 attempts in total, and none of the 29 configurations exceeding 30%; DeepSeek V4.1 Flash, Qwen 3.8 Max, and Kimi K3.1 Pro did not solve a single question correctly, and the rule induction category had only 5 correct out of 870 attempts.

Generated by AI from reports · Updated 1 days ago

Progress timeline

Nov 17
  1. WeChat · Synced (Jiqizhixin)
    Einsia AI releases Visual Cipher Bench: the strongest, GPT-6 Astra, answers only 26.7% correctly

    Einsia AI's Navers Lab released Visual Cipher Bench, an agent visual reasoning benchmark. Across 3480 attempts by 8 frontier models, the strongest configuration, GPT-6 Astra (max), answered only 26.7% correctly. The benchmark contains 40 original visual puzzles and 67 black-and-white line drawings, with clues drawn only in the images, divided into four categories: spatial manipulation and simulation, logical constraints and planning, rule induction, and knowledge decoding. None of the 29 reasoning-intensity configurations exceeded 30%; DeepSeek V4.1 Flash, Qwen 3.8 Max and Kimi K3.1 Pro each got not a single question right; in the rule induction category, only 5 of 870 attempts were correct. The research team additionally took 10 questions and, after adding one core-rule hint for each of three GPT-6 models, overall accuracy rose from 12.2% to 51.1%.

Nov 12
  1. arXiv
    Visual Cipher Bench: evaluating long-horizon visual reasoning in LLM agents

    Researchers introduce Visual Cipher Bench, which uses 40 original visual puzzles to evaluate long-horizon visual reasoning in LLM agents; the puzzles cover four categories: spatial manipulation and simulation, logical constraints and planning, rule induction, and knowledge-based decoding, each providing only one to three black-and-white line drawings and brief submission instructions, and the agent must discover the solving mechanism entirely from the images. Across 29 reasoning effort configurations of 8 frontier models, the strongest, GPT-6 Astra (max), achieved only 26.7% accuracy, and 5 of the 8 models were below 1%. On a subset of 10 tasks, a non-spoiler mechanism hint raised accuracy from 12.2% to 51.1%. The authors therefore conclude that the main bottleneck of current agents is not executing multi-step plans, but autonomously discovering rules from visual evidence.

Related discussion

Total 5 items

Attention trend

New sources per day

5 in 4 days·most 3 (Nov 17)·today 0

  • Nov 17: 3 new sources (1 outlets, 2 accounts)
  • Nov 18: 2 new sources (0 outlets, 2 accounts)
  • Nov 19: 0 new sources (0 outlets, 0 accounts)
  • Nov 20: 0 new sources (0 outlets, 0 accounts)

Hourly attention

No comparable trend data