AgentLAB: A Benchmark for Long-Horizon Attack Evaluation of LLM Agents
AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks
Quick read
Based on 论文全文 （AI 生成）; refer to the original
The researchers release AgentLAB, a benchmark for evaluating the vulnerability of LLM agents under long-horizon attacks, comprising 5 attack types—intent hijacking, toolchain, task injection, goal drift, and memory poisoning—covering 28 agent environments and 644 safety test cases. The authors state this is the first agent safety benchmark specifically dedicated to evaluating long-horizon adaptive attacks. Initial evaluations on GPT-4o, GPT-5.1, Gemini-3, Claude-4.5, Llama-3, and Qwen-3 show that most agents remain highly vulnerable; the paper reports that GPT-5.1's average attack success rate exceeds approximately 70%, while Claude-4.5 overall is 28.9%, with 0% for task injection. Defenses designed for single-turn attacks, such as Self-Reminder, Llama-Guard, Repeated Prompt, and DeBERTa Detector, struggle to transfer reliably to long-horizon scenarios. The benchmark is publicly available at https://tanqiujiang.github.io/AgentLAB_main。
- LLM agent 越来越多地部署在长时程、复杂环境中，现有 agent 安全研究主要关注单轮或静态场景，无法刻画攻击者通过多轮用户–agent–环境交互逐步达成目标的威胁。作者认为这类长时程攻击是现实且更危险的风险，需要专门的评测基准。
- 作者提出 AgentLAB，一个面向长时程攻击的 LLM agent 安全基准，包含意图劫持、工具链、任务注入、目标漂移、记忆投毒五类攻击，覆盖 28 个真实工具型环境和 644 个安全测试用例。其核心是用多 agent 框架（planner、attacker、judge，部分含 verifier）实现自适应、多轮的攻击生成与优化。
- 在 GPT-4o、GPT-5.1、Gemini-3、Claude-4.5、Llama-3、Qwen-3 等 agent 上评测，作者发现它们普遍对长时程攻击高度脆弱，GPT-5.1 平均 ASR 超过约 70%；长时程注入明显优于单轮注入，而 Self-Reminder、Llama-Guard、Repeated Prompt、DeBERTa 检测等单轮防御难以可靠迁移。
- 作者称这是初步评测，防御方面只测试了少量单轮防御，结论是它们难以适配长时程场景，呼吁设计针对性防御；基准本身被定位为可扩展的 live benchmark，便于后续加入新环境、攻击类型和防御。
The paper reports attack success rates for five types of long-horizon attacks across 28 environments and 644 use cases, and shows that single-turn defenses are difficult to transfer.