A mechanistic localization study of prompt injection compliance decisions
Start here
Two studies used causal methods to locate the decision layers where models comply with prompt injection instructions. The first study performed layer-by-layer causal activation patching on five models ranging from 4B to 32B, finding that attack information can be linearly decoded from the first layer onward, but only a late-layer bottleneck in the final third of the network has a clear causal effect on behavior; patching this bottleneck reversed the model's compliance with attack instructions in the tested cases, the relevant mechanism lies in a compact linear subspace, and different model families exhibit similar structures. The second study analyzed the gap between role signals and behavior changes in indirect prompt injection through counterfactual role probes, component-wise activation patching, and interventions on AgentDojo trajectories: role decoding persists under content and format changes and, in controlled Qwen tests, precedes the tool selection effects of patching along independently estimated role directions; on AgentDojo, directions estimated from hijacked and resistant training trajectories reduced attack success rates at pre-action and injection-segment positions, but had little effect at random positions; in longer Qwen trajectories, single-point edits had weaker effects in later layers, and cross-segment and repeated edits could, on the same evaluation set... (the report content is truncated here). Both studies are mechanistic investigations within the tested models and experimental settings, not general defense guarantees for arbitrary models.
Generated by AI from reports · Updated 4 days ago
Latest developments
Progress timeline
- arXiv：越狱、提示注入、投毒与防御The study uses causal tracing to distinguish readable signals from effective interventions in indirect prompt injection.
The study analyzed the gap between role signals and behavior changes in indirect prompt injection through counterfactual role probes, component-wise activation patching, and interventions on AgentDojo trajectories. Role decoding persists under content and format changes and, in controlled Qwen tests, precedes the tool selection effects of patching along independently estimated role directions. On AgentDojo, directions estimated from hijacked and resistant training trajectories reduced attack success rates at pre-action and injection-segment positions, but had little effect at random positions. In longer Qwen trajectories, single-point edits had weaker effects in later layers, and cross-segment and repeated edits reduced attack success rates on the same evaluation set. Removing the learned channel subspace preserved role decoding, but effective intervention directions transferred poorly across test channels.
- arXiv：越狱、提示注入、投毒与防御The study uses causal activation patching to locate the decision layers where models comply with prompt injection instructions.
The researchers performed layer-by-layer causal activation patching on five models ranging from 4B to 32B and found that attack information can be linearly decoded from the first layer onward, but only a late-layer bottleneck in the final third of the network has a significant causal effect on behavior. The authors report that patching this bottleneck can reverse the model's compliance with attack instructions in the tested cases; the relevant mechanism lies in a compact linear subspace, and different model families exhibit similar structures. This is a mechanistic study within the tested models and experimental settings, not a general defense guarantee for arbitrary models.