ICON：基于推理时纠正的 Agent 间接提示注入防御

Adaptive indirect prompt injection (IPI) embedded in retrieved content hijacks an agent's tool calls, while refusal-based defenses interrupt tasks that should still continue.

• Scenario: The authors argue that once an agent connects to external data and APIs, retrieved content such as emails and web pages can carry malicious instructions. Unlike direct jailbreaks, IPI can force the model to execute unauthorized tool calls and bypass traditional safety guardrails.
• Existing shortcomings: The authors state that template-based and post-hoc tool filtering rely on surface patterns and can be bypassed by adaptive variants; safety fine-tuning tends to over-refuse, prematurely terminating complex agent loops, and retraining costs are unsuitable for dynamic deployment. In Table 1, Template and Tool-Filter have ✗ for Security and Utility, Fine-tuning has ✗ for Utility and Efficiency, and ICON has ✓ for all three.
• Core observation: The authors state that IPI that successfully triggers a specific adversarial intent causes attention to abnormally concentrate on a small number of injected tokens; benign trajectories require integrating long-range context.
• Approach: Proposes ICON (Inference-Time Correction): uses latent-space traces to detect attacks, then uses attention steering to pull the trajectory back to the original task rather than aborting execution.
• Threat model:
  • Victim system: Tool-augmented agent M with tool set F. The trajectory is St = (Iu, A1:t−1, O1:t−1); actions are tool calls, and observations are execution results or retrieved content.
  • Attacker knowledge: Grey-box. Cannot access the target LLM's parameters or architecture but can infer context through an intermediate server. The authors state this aligns with Zhu et al., 2025 and Anonymous, 2025.
  • Capabilities and goals: Can only inject payload δ into the observation stream Ot; can use partial task knowledge to make δ semantically consistent with benign content; the goal is to lead to unauthorized tool call fadv while maintaining a covert latent trajectory.
  • Defender: System operator with white-box access to the local LLM, able to inspect and intervene in hidden states and attention. The goal is to detect IPI traces in St ⊕ δ and mitigate their impact at inference time.

Indirect prompt injection attacks

• Greshake et al., 2023, Wu et al., Zhan et al., 2025b: place malicious payloads in external sources to hijack the agent's tool use. The paper accordingly separates IPI from direct prompt injection where users directly bypass internal guardrails.
• Willison, 2022, Perez et al., 2024, Zhan et al., 2024: use template-based prompts to make the LLM execute adversary instructions. The introduction also cites Zhan et al., 2024, Liu et al., 2025, Zhan et al., 2025a, stating that adaptive IPI is increasingly hard to distinguish from benign task context.
• Liu et al., 2025, Anonymous, 2025, Zhang et al., 2024: use strong LLMs or gradients to generate universal attack prompts. The authors state that adaptive attacks can dynamically bypass static guardrails that remain effective against ordinary attacks.

Prompt injection defenses

• Willison, 2022 and San, 2023: use delimiters to isolate untrusted content, or restate system instructions each round.
• ProtectAI's Deberta detector (2023), MELON (Zhu et al., 2025), perplexity-based detection (Jain et al., 2023): intercept at the input stage. The authors call them post-filters, which often trade off utility against safety and introduce latency.
• Llama-Guard (Llama Team, 2024), Qwen-Guard (Zhao et al., 2025): fine-tuned safety guardrails. The authors state they require large-scale high-quality safety data and are computationally too expensive for many agent applications.

Inference-time probing and steering

• Zhang et al., Todd et al., as well as induction heads (Crosbie & Shutova, 2025), successor heads (Gould et al.): different attention heads serve different functions.
• Attention Tracker from Hung et al., 2025, Xia et al., 2025: use internal activations for diagnostic classification. Lookback Lens from Chuang et al., 2024 is categorized as an interpretability-driven method; latent-space manipulation is described as modifying hidden states or attention at inference time without changing parameters.
• LM-Steer (Han et al., 2024): uses a steering matrix to linearly transform the frozen model's output embeddings to modulate generation attributes without additional training. The paper does not list it as an experimental baseline.

Baselines and benchmarks: Experimental comparisons include Repeat Prompt (Debenedetti et al., 2024), Delimiting (Willison, 2023), MELON, and Qwen3Guard on the text side (LLM-Detector in Table 2) and Gemini on the multimodal side (LLM-Detector in Table 5). Evaluations use InjectAgent, AgentDojo; training uses TrojanTools; vision tasks use Visual Prompt Injection Benchmarks (Wan et al., 2024).

The authors place ICON after the above work: static prompts and external detection struggle to balance utility and safety, fine-tuned guardrails have high data and computational costs; ICON aims to achieve what the authors call a better balance among safety, efficiency, and utility.

ICON has two phases: offline synthesis of adaptive IPI near the benign boundary for probe training and rectifier calibration; online detection of over-focusing by the Latent Space Trace Prober (LSTP), followed by attention redirection by the Mitigating Rectifier (MR) without updating model weights.

Problem Setting and Objectives

• Formalization follows Yao et al., 2022 and Zhu et al., 2025. The user instruction is Iu, and the trajectory at time t is St = (Iu, A1:t−1, O1:t−1).
• The attacker seeks a context-dependent δ that minimizes the difference D between the compromised and benign trajectories while ensuring M(St ⊕ δ) = fadv. D is analyzed for stealthiness by the LLM-as-Optimizer described in the paper.
• The defender simultaneously minimizes detection loss and maximizes the log-probability of the correct task output after rectification. The paper writes min_ϕ,γℒ(h_ϕ,y)−log P_M(y∣ R(S_t⊕δ;γ;τ)), optimizing over probe parameters ϕ and steering intensity; τ appears in the rectification operator R but not in the minimization subscript.
• Intervention occurs on layer representations when processing compromised observations and generating the next action At, without modifying model weights.

Offline Boundary Samples

• The synthesis module aims to generate boundary samples satisfying the attacker's objectives above. The authors state the purpose is not to study the attack itself but to delineate the boundary between adaptive and benign.
• Contextual Tool Alignment: The optimizer selects the most plausible malicious tool fadv from F given the current trajectory, to avoid logical inconsistencies that might trigger surface-level semantic filtering.
• Iterative Stealthiness Optimization: Starting from random initialization, δ is rewritten based on feedback from the agent's previous successes or failures, reducing D while still satisfying the success constraint.
• The authors claim this enables LSTP and MR to learn over-focusing signals rather than memorizing keywords. The paper does not specify the optimizer LLM's name, iteration count, or payload templates.

Probing: FIS and LSTP

• Focus Intensity Score (FIS) measures the dependence of the entire generated sequence Y on the context St. The authors distinguish it from approaches that only look at the last input token or the first generated token (Hung et al., 2025).
• For layer l, head h, and generation position i, a length-normalized token entropy is computed; the head score is 1 minus the average entropy over the sequence, and the layer score is the average over heads. ε is only described as a numerical stability term; the paper does not provide experimental values.
• The authors argue that low-entropy heads are more likely to capture task-critical patterns and are more susceptible to exploitation in IPI. FIS is used both to select layers for LSTP and heads for MR.
• LSTP compresses each head's entropy sequence into three dimensions: minimum entropy, average entropy, and standard deviation. It selects K sensitive layers per layer FIS and concatenates them. The paper gives an example of 384-dimensional features for H=32, K=4, but does not state whether this was used in experiments. The classifier is a 1D-CNN plus MLP, followed by Adaptive Global Maximum Pooling, making detection independent of response length.

Mitigation: Mitigating Rectifier

• The authors argue that refusal vectors (Arditi et al., 2024; Wu et al., 2025) often prematurely terminate the execution loop. MR instead suppresses anomalous query-key dependencies and uses softmax reallocation to push weights toward positions masked as 0.
• Scope τ: For heads flagged by FIS, the threshold is the (1−τ)th percentile of that head's attention weights; positions not below the threshold enter the mask.
• Intensity γ: Before softmax, \tilde{A}_{l,h}=A_{l,h}\odot[1+M_{l,h}\cdot(\gamma-1)]. The paper states γ<1 acts as a suppression coefficient. The paper does not report the γ, τ, or number of anomalous heads used in experiments.

Experimental Setup

• Tested backbones: Text: Qwen-3-8B, Llama-3.1-8B, Mistral-8B; multimodal: Qwen3VL, InternVL, MiniCPM. All connected to ReAct. The main text writes the vision models as Qwen-VL, InternVL, MiniCPM.
• Data: Evaluation: InjectAgent, AgentDojo; training: TrojanTools, which Table 2 also lists as an attack setting. Visual tasks use the Visual Prompt Injection Benchmarks from Wan et al., 2024. The paper does not report per-set sample sizes.
• Attack settings: Table 2 lists Ignore Instruction, Combined Attacks, TrojanTools. The paper does not elaborate on the construction of these three in the experimental section.
• Baselines: Repeat Prompt, Delimiting, MELON; the text LLM-Detector is Qwen3Guard; the multimodal LLM-Detector is Gemini. The authors state that Gemini is used to provide an upper bound on performance.
• Metrics: Lower ASR is better for the defense, higher UA is better for the defense. Table 3 additionally reports ADR and URR, with standard deviations from 5-fold cross-validation. The table contains BU, which the paper does not define. The paper does not specify the decision rules or review models for ASR and UA, nor the number of repetitions for Table 2 and Table 5.
• Training scale: In Table 4, ICON is under 2 minutes, 255 samples, 31,553 parameters. The introduction also states a few hundred samples and convergence in one minute; Section 5.2.1 states about 31k parameters.

Main Results

The table below is the Avg row of Table 2, in %. LLM-Detector is Qwen3Guard.

Swipe horizontally to read the table

Model	No Defense ASR/UA	Repeat Prompt ASR/UA	LLM-Detector ASR/UA	ICON ASR/UA
Qwen-3-8B	32.7 / 43.1	16.8 / 51.0	0.2 / 43.2	0.4 / 53.8
Llama-3.1-8B	35.7 / 20.7	21.3 / 28.1	0.0 / 20.9	0.3 / 29.7
Mistral-8B	30.9 / 10.7	17.8 / 15.2	0.0 / 10.7	0.3 / 15.9

• The authors state that ICON has an average ASR of 0.4%, comparable to LLM-Detector's 0.2%, and claim this comes from a lightweight probe rather than an external guardrail. These two numbers are consistent with the Avg row for Qwen-3-8B; the other two backbones have an ICON average ASR of 0.3% and LLM-Detector of 0.0%. The abstract also writes competitive 0.4% ASR.
• The authors state an average 10% UA improvement over other defenses, and that Qwen-3-8B's UA is 53.8%, with the next-best defense below 49%. In Table 2, that model's Repeat Prompt average UA is 51.0%. There are also two cells where Repeat Prompt's UA is higher than ICON's: Qwen Combined Attacks is 53.1% vs 52.7%, and Mistral Ignore Instruction is 17.8% vs 17.6%.
• The authors state a further 3% ASR reduction and 10% UA improvement over MELON, because MELON cannot rerun inference after triggering; they also believe template methods have some effect on Ignore Instruction but are weaker on TrojanTools, and slightly increase UA on Llama and Mistral, possibly because structured prompts help these models parse intent. On TrojanTools, ICON's ASR is higher than LLM-Detector's: 1.2% vs 0.6%, 0.8% vs 0.0%, 0.9% vs 0.0%. MELON's Avg is Qwen 3.2/43.7, Llama 1.6/20.9, Mistral 1.1/11.0; the paper does not specify which model the 3% and 10% correspond to.

OOD Transfer

• The probe is trained only on TrojanTools, then tested on InjectAgent and AgentDojo. Table 3 gives ADR, URR, and standard deviations from 5-fold cross-validation; Mistral-8B is not included. The no-defense columns are all 0%.
• Qwen-3-8B: on InjectAgent, ADR 80.1%±1.9, URR 62.3%±2.3; on AgentDojo, 98.0%±1.2, 69.6%±2.7. Llama-3.1-8B: 85.4%±1.3, 47.6%±3.1, and 97.3%±1.8, 55.9%±3.3.
• The authors state that the detection rate on AgentDojo exceeds 97% and on InjecAgent exceeds 80%, and that both backbones have utility recovery rates above 60%. The latter sentence is inconsistent with Llama's two URR cells. The authors call this zero-shot transfer, with the probe capturing latent traces rather than training semantics. The introduction states that on AgentDojo the ASR is lower than Qwen3Guard's, and gives a 69% average Utility improvement; Table 3 has no Qwen3Guard column, and the paper does not explain whether 69% refers to 69.6%±2.7.

Multimodal

Cells are ASR / UA (%), per Table 5. The header arrows are written as ASR↑, UA↓, opposite to the direction in Section 5.2. BU is Qwen3VL 70.5, InternVL 64.5, MiniCPM 72.0, average 69.0, undefined in the paper.

Swipe horizontally to read the table

Model	No Defense	Repeat Prompt	Gemini	ICON
Qwen3VL	69.9 / 21.3	47.8 / 46.5	0.0 / 21.9	3.8 / 44.9
InternVL	47.4 / 33.9	41.6 / 26.5	0.0 / 34.1	2.6 / 47.8
MiniCPM	59.7 / 29.0	59.6 / 22.0	0.0 / 30.3	2.4 / 48.8
Avg	59.0 / 28.1	49.7 / 31.7	0.0 / 28.8	2.9 / 47.2

• The authors state that ICON has an average ASR of 2.9%, a large drop from Repeat Prompt's 49.7%; Gemini's ASR is 0, with an average UA of 28.8%, while ICON's is 47.2%, which the authors describe as nearly a 20% absolute improvement over Gemini.
• On Qwen3VL, Repeat Prompt's UA is 46.5%, higher than ICON's 44.9%. On InternVL, Repeat Prompt's UA is 26.5%, lower than the no-defense 33.9%.
• The introduction states a 42% average Utility recovery for multimodal. Table 5 has no such column.

Efficiency and Attention Maps

• Table 4: Qwen3Guard is over 10 hours, 1.19M, 8B; ICON is under 2 minutes, 255, 31,553; MELON and Repeat Prompt have training cost, data volume, and parameter count all 0. The authors state that ICON approaches the training overhead of heuristic methods while having learned latent-trace detection. The paper does not report inference time.
• The text in Figure 3's plots is Head 21 of Layers 22, 23, 24, with the x-axis Key Position and y-axis Query Position, ticks up to 350, and a color scale of about 0 to 0.015. The main text writes Qwen-3-8B head 21, layers 22-25.
• The authors state that on the attack side, attention collapse appears around positions 100 and 300, corresponding to the malicious payload; after MR, the abnormal peaks are suppressed and attention spreads to a broader trajectory. The transcribed text cannot show hotspot positions, so only the authors' description and the text in the figures can be recorded.

Other Ablations and Analyses

• The paper does not report ablations of γ, τ, the number of selected layers K, the number of abnormal heads, learning rate, or number of training epochs.
• Figure 1's axes are ASR (ticks 0–40) and UA (ticks 35–55), with legend entries No Defense, Repeat Prompt, Delimiting, MELON, LLM-Detector, Ours; the main text does not give coordinates for each point. The caption states that ICON has lower ASR and higher UA.
• Table 2's BU is Qwen-3-8B 69.8, Llama-3.1-8B 34.5, Mistral-8B 20.1; the paper does not define it or compare it with UA.

Limitations and Future Directions Noted by the Authors

The paper does not specifically discuss limitations. Neither the Conclusion nor the Impact Statement mentions shortcomings, unfinished items, or follow-up experiments.

Experimental Coverage

• Text experiments use Qwen-3-8B, Llama-3.1-8B, and Mistral-8B in ReAct, with attack settings of Ignore Instruction, Combined Attacks, and TrojanTools (Section 5.1, Table 2).
• Multimodal experiments use Qwen3VL, InternVL, MiniCPM, and Visual Prompt Injection Benchmarks; baselines are Repeat Prompt and Gemini (Table 5). Text baselines also include Delimiting, MELON, and Qwen3Guard (Table 2).
• OOD is reported only for Qwen-3-8B and Llama-3.1-8B; training data is TrojanTools, testing is on InjectAgent and AgentDojo, with ADR and URR accompanied by standard deviations from 5-fold cross-validation (Table 3).
• Efficiency comparisons include training time, data volume, and parameter count for Qwen3Guard, MELON, Repeat Prompt, and ICON (Table 4). The attention map is for Head 21 of Qwen-3-8B (Figure 3, Section 5.4).
• The paper does not report the sample size for each benchmark, the rules for determining ASR and UA or the review model, the number of repetitions for Table 2 and Table 5, the values of γ and τ, or the name of the optimizer LLM used for the synthetic module.

ICON detects IPI attention over-focusing at inference time and steers, avoiding refusal that terminates agent trajectories.

• Problem: Malicious instructions in retrieved observations can steer the policy toward unauthorized tool calls fadv. The authors believe templates and filters are easily bypassed by adaptive attacks, while fine-tuning over-refuses.
• Method: The attacker is grey-box; the defender has white-box access to the local LLM. FIS measures attention entropy; LSTP uses minimum entropy, mean, and standard deviation for detection; MR uses τ to define a mask before softmax and γ to suppress abnormal query-key dependencies. Training uses TrojanTools synthetic samples.
• Main results: Averaged over the three attacks in Table 2, ICON achieves ASR of 0.4%, 0.3%, 0.3% and UA of 53.8%, 29.7%, 15.9% on Qwen-3-8B, Llama-3.1-8B, and Mistral-8B, respectively. In the same table, Qwen3Guard has ASR of 0.2%, 0.0%, 0.0% and UA of 43.2%, 20.9%, 10.7%. On the TrojanTools row, ICON's ASR is higher than that detector.
• Transfer and multimodal: When trained only on TrojanTools, Qwen-3-8B achieves ADR of 98.0%±1.2 and URR of 69.6%±2.7 on AgentDojo (Table 3). Across three MLLMs, ICON has an average ASR of 2.9% and UA of 47.2%, while Gemini has 0.0% and 28.8% (Table 5).
• Authors' conclusion: The authors state ICON better balances safety, utility, and efficiency. In Table 4, ICON uses less than 2 minutes, 255 samples, and 31,553 parameters, while Qwen3Guard uses over 10 hours, 1.19M samples, and 8B parameters. Utility claims are over 50% task utility gain in the abstract, 69% average Utility improvement and 42% average Utility recovery in the introduction, and over 50% higher utility than commercial-grade baselines in the conclusion.