SafeAgentBench: a benchmark for safety task planning in embodied LLM agents

SafeAgentBench is a benchmark for safety task planning in embodied LLM agents. Testing on eight agent architectures shows that their refusal rate for dangerous tasks is only about 3% to 10%. The benchmark contains 750 tasks covering 10 categories of danger, divided into three types: explicit dangerous tasks, deceptive dangerous tasks, and normal tasks with hidden dangers, and it builds the SafeAgentEnv simulation environment based on AI2-THOR, supporting 17 high-level actions. The evaluation uses two sets of metrics at both the execution level and the semantic level: the former checks whether the agent actually executes dangerous actions, and the latter checks whether its plan contains dangerous steps. Results show that changing the underlying LLM has little impact on safety performance, deceptive phrasing can effectively bypass refusal, and planning frameworks with a safety reflection step provide only marginal improvement; some agents planned unsafe actions but failed to execute them due to environmental constraints, and the authors argue that semantically unsafe planning should still be treated as a real risk signal.