Prompt injection is role confusion
Prompt Injection as Role Confusion
Quick read
Based on 论文全文 （AI 生成）; refer to the original
The researchers propose the role confusion theory, arguing that large language models judge content provenance by text style rather than role labels, and design role probes to measure role perception inside the models. Based on this, they propose the CoT Forgery attack, which injects forged reasoning into user prompts or tool outputs, achieving a 60% attack success rate on multiple frontier models on StrongREJECT, with baselines near zero. Across 1000 agent data exfiltration attempts, the attack success rate rises from 2% in the lowest confusion decile to 70% in the highest decile as measured by the probe. Style ablations show that removing the reasoning style drops the attack success rate from 61% to 10%.
- LLM 通过 <user>、<tool> 等角色标签建立权限边界，但 prompt injection 仍普遍成功。作者认为根因是模型并非按标签判断文本来源，而是按风格等可伪造线索推断角色，导致低权限文本冒充高权限角色。
- 训练 role probes：把相同内容包进不同角色标签，使探针只能学到标签的几何特征，再测量模型内部把每个 token 感知成哪个角色。提出 CoT Forgery 攻击，把伪造的推理风格文本注入 user 或 tool 通道，并用 destyling、明文声明角色等消融验证风格与角色感知的因果关系。
- CoT Forgery 在 StrongREJECT 上对多个前沿模型达到约 60% 攻击成功率，基线接近零；agent 数据外泄任务中标准注入多失败，CoT Forgery 升至 56–70%。destyling 使成功率从 61% 降到 10%。探针显示被注入文本在潜空间中落入其冒充角色的表示空间，且 role confusion 程度在生成前就能预测攻击成功：CoTness 最低分位 9%、最高 90%；agent 注入的 Userness 最低分位 2%、最高 70%。
- 探针只覆盖 4 个 20–120B 模型，扩展到更大模型是未来工作；线性探针假设角色占据方向性子空间。作者提出可继续研究：用 role probes 评估防御是否真正重塑表示、其他注入家族是否统一于 role confusion、如何实现潜空间中的干净角色分离、以及生成前检测角色错配。
The paper attributes prompt injection to models judging role provenance by style rather than labels, and its role probe method can be used to test whether defenses truly change internal representations.