Claude Fable 5.2 solved 80% of the 391 core tasks in the embodied robot evaluation in open-book mode, and only 47% in closed-book mode
Fable 5.2开卷80%,闭卷仅47%!最强编程智能体遇上机器人难题
Researchers from 8 universities including Northwestern University and Carnegie Mellon University propose the embodied evaluation framework Embodied First Exam (EFE), which connects coding agents to simulated robots, covering 1156 tasks and 47 kinds of robots; the paper is arXiv 2611.12177. On the same batch of 391 core tasks, the strongest Claude Fable 5.2 privileged mode (reading the full simulation state, unlimited resets) solved 80%, while the standard mode (relying only on sensors, no resets by default) solved only 47%; of 456 core tests in standard mode, 169 were not solved by any agent. A control experiment on GPT-6 Astra shows that adding unlimited resets narrows the gap with privileged mode more than adding precise state. The team used coding agents to annotate 6027 failed attempts by 10 agents and found that different agents often fail at the same stage; except for real-robot demonstrations, all results come from simulation, and each test was run only once.
The evaluation gives the gap in success rates between privileged and standard modes for coding agents completing robotic tasks; the per-stage failure annotation method can be transferred to other agent evaluations.