Agents at risk: how users can inadvertently undermine LLM safety
Agents at Risk: How Users Unwittingly Undermine LLM Safety
Quick read
Based on 论文全文 （AI 生成）; refer to the original
The paper proposes the UReCoM (User-Rephrased Context Manipulation) attack, in which an attacker induces ordinary users through public posts and other content to copy adversarial payloads into requests to LLM agents, causing external content to enter the agent as user task context. The authors compare five prompt injection baselines on an extended AgentDojo benchmark, and UReCoM achieves higher attack success rates under preventive defenses such as Sandwich, StruQ, and SecAlign, and detection-based defenses such as Perplexity, DataSentinel, and CausalArmor. On GPT-5, all five baseline attacks had a success rate of 0.00%, while UReCoM reached 68.57%; DeepSeek-V3 was 70.97%, and Llama 3.1 8B was 62.66%. Across 12 commercial agents (6 travel planners, 6 web operators), when no safety request was present by default, the environment constraint bypass rate for travel planning agents ranged from 83% to 100%, and user-rephrased links and promo codes were checked only when users explicitly requested verification.
- LLM-based agents（如行程规划、网页操作类）容易被外部对抗内容当作任务约束，即 context confusion。已有防御假设恶意内容来自可识别的外部检索源，但当攻击者诱导用户把对抗内容转发进用户请求时，这一假设失效。
- 提出 User-Relayed Context Manipulation（UReCoM）攻击：攻击者不直接接触 agent，而是用社交媒体等内容诱导良性用户把对抗载荷（如假优惠码、链接、客服联系方式）复制进自己的请求。作者基于 AgentDojo 构造基准，用 Gemini 2.5 生成 instruction-style、tips-style、promotion-code-style 三类载体，并在不同放置顺序下测试五类 prompt-injection 基线和多种防御。
- 在六种防御下，UReCoM 的 ASR 普遍高于五个 prompt-injection 基线；载体放入用户输入后更高，载体在前（I+U）尤其有效。在 Claude、GPT、Llama、DeepSeek 等模型上，UReCoM 的 ASR 也远高于基线，GPT-5 对基线为 0% 但对 UReCoM 仍有 68.57%。在 12 个商用 agent 上，默认无安全请求时行程类 agent 的环境约束绕过率为 83%–100%，显式要求安全验证后明显下降；网页类 agent 会按恶意 URL 导航并执行点击、输入等操作，仅在沙箱阻断时停止。
- 作者观察到商用 agent 并非没有安全能力，而是仅在用户显式要求时才调用，默认不主动验证用户转发的任务实体。作者据此提出需要默认的、provenance-aware 的验证与风险感知停止规则，以及 task-entity 级别的防护。
论文提出用户转述式上下文操纵攻击，并给出在 6 个模型与 12 个商用智能体上的攻击成功率，可供智能体部署方重新审视用户输入通道的信任假设。