WorldFailBench: Do World Models Understand Policy Failures?

The researchers propose the WorldFailBench benchmark, which uses three tasks to test whether world models can faithfully simulate how robot policies fail; experiments on seven world models show a clear gap in their understanding of policy failures. Recent benchmarks mostly examine the potential of world models as policy evaluators through outcome-based evaluation of task success or failure, but this cannot confirm whether a model faithfully reproduces the failure process; distorted failure presentation may lead to misdiagnosing policy weaknesses, making a policy appear deployable when it fails under small scene changes. The three tasks examine failure cause (whether the policy's observable errors can be reproduced), failure stage (whether failures concentrate at the same stage of full task execution), and failure boundary (the conditions under which the policy transitions from success to failure under controlled scene changes). The benchmark covers manipulation tasks in the LIBERO and RoboTwin simulation environments and real manipulation tasks on the G1 humanoid robot; the authors state that the experimental results can guide the development of more reliable world models for policy evaluation.