TRACER: Trajectory-regularized training against indirect prompt injection

1. 01What problem does this paper attempt to solve?

   The authors state that changing the channel for the same payload changes ASR, and single-package training cannot be assumed to transfer by default.

   The attack success rate of the same malicious payload varies with the presentation channel, and robustness trained in a single packaging cannot be assumed to transfer to other channels.

   • Scenario: RAG, tool assistants, and multi-agent systems must handle text the model did not write and cannot fully trust. The authors call malicious instructions hidden in external content, rather than in the direct user prompt, indirect prompt injection.
   • Existing shortcoming: the authors state existing defenses are usually trained and evaluated within a single wrapper, so "does robustness survive a channel change" is rarely isolated directly.
   • Observation: six intermediate snapshots trained only on follow-up-style attacks, when the same attack string is repackaged as a retrieved document, raise the average ASR from 18% to 53%. The authors call this a framing gap, and call follow-up being weaker than documents, tool outputs, or agent messages an authority gradient. The paper states authority is only an explanatory shorthand, not an isolated causal conclusion; format, position, source marking, and instruction-tuning priors may also contribute. The controlled conclusion is: with the payload fixed, channel rendering changes ASR.
   • Method: propose TRACER (Trace-Regularized Adversarial Channel Exposure with Reward-shaping). At training time, sample presentation wrappers, use answer anchoring and evidence-gated rewards, and when injected evidence is invalid, use ATRC to align attacked reasoning to the clean anchor.
   • Threat model:
     • Capabilities: the attacker can only modify untrusted external context, not the original user task, hidden system or developer instructions, model weights, training process, or evaluation harness.
     • Defense task: relevant and valid external context should be treated as evidence; adversarial text within it should not be treated as instructions that must be followed. The paper calls this semantic-control.
     • Channels: C = {followup, rag, tool, agent}, corresponding to retrieved documents, tool outputs, other agent messages, and user follow-ups.
     • Judgment: conditional ASR counts only samples answered correctly cleanly and incorrectly after injection, to distinguish injection failure from ordinary mistakes.

   Collapse this answer

2. 02What related research exists?

   The paper situates itself alongside indirect injection benchmarks, RAG poisoning, inference-time defenses, and verifiable rewards.

   Indirect prompt injection and benchmarks

   • Greshake et al. (2023): introduced indirect prompt injection as a practical threat in LLM-integrated applications.
   • Subsequent benchmarks: BIPIA, InjecAgent, AgentDojo, ASB, WASP, Hidden-in-Plain-Text (Yi et al., 2025; Zhan et al., 2024; Debenedetti et al., 2024; Zhang et al., 2025; Evtimov et al., 2025; Guo and Wei, 2026) extend the attack surface to agents, dynamic environments, and web pages. The authors state this paper does not build new environments but isolates the packaging factor common to these settings.

   RAG robustness and attack design

   • Conflicting or poisoned retrieval: ConflictQA, RGB, PoisonedRAG (Xie et al., 2024; Chen et al., 2024; Zou et al., 2025) evaluate failures under conflicting or corrupted retrieval.
   • TopicAttack (Chen et al., 2025c): studies indirect injection closer to real-world scenarios. The authors state this line of work mostly treats robustness as a property of a fixed pipeline; this paper fixes the payload and only changes the presentation channel.

   Defenses

   • Separation, marking, and alignment training: StruQ uses structured instruction fine-tuning for prompt/data separation (Chen et al., 2025a); Hines et al. (2024)'s spotlighting does data marking at inference time; SecAlign uses preference optimization for alignment-style training (Chen et al., 2025b).
   • Runtime modules: MELON, CaMeL, instruction detection, robustness-via-referencing, PromptArmor, tool-result parsing, ICON, AttriGuard. The authors state these methods typically harden a fixed pipeline or add inference-time modules; this paper focuses on training the base model so that it maintains correct post-attack behavior across different untrusted channels.

   Reasoning reinforcement learning and evaluation agenda

   • Verifiable rewards: GRPO (Shao et al., 2024), DeepSeek-R1 by Guo et al. (2025), DAPO by Yu et al. (2025), and stepwise process rewards (Lightman et al., 2024; Wang et al., 2024). The authors state these methods improve reasoning on clean inputs; TRACER treats clean trajectories as anchors that must remain consistent when evidence is invalid.
   • Evaluation standards: Jia et al. (2025) and Zhan et al. (2025) argue defenses should consider both safety and utility, and target adaptive attacks rather than only static strings. The authors state the framing-gap analysis fits this agenda: reporting only single-channel scores can miss failures after repackaging.

   Baselines and benchmarks: Experimental comparisons include the base model without LoRA, answer-only channel exposure, the composite reward without ATRC, and ablations of the RAG wrapper and RAG-specific system prompt. The main task is GSM8K; cross-domain evaluation of RAG framing is also done on Natural Questions and TriviaQA. The paper does not run the above published defenses as experimental baselines.

   The authors position this paper as complementary to these benchmarks and inference-time defenses: it introduces no new environments and does not replace runtime modules, but writes the packaging channel into the training objective and reports cross-framing transfer.

   Collapse this answer

3. 03How does the paper address this problem?

   TRACER samples channels, uses evidence-gated rewards and ATRC, and preserves clean anchors under invalid injection.

   TRACER treats the presentation channel of untrusted context as a training random variable. Each sample first answers cleanly, then the same adversarial payload is sent back through a sampled channel, and reward is given only to the post-attack answer; when evidence is invalid, ATRC pulls the clean reasoning span.

   Problem setup

   • Objects: task is (x_i, y_i), a_i is adversarial content, c ∈ C = {followup, rag, tool, agent}. ρ_c(a_i) changes only rendering, not the payload.
   • Three quantities: clean accuracy Acc^0, post-attack accuracy Acc^c on channel c, and conditional attack success rate. A positive transfer gap means suspicion learned on the training channel does not fully transfer to the evaluation channel.
   • Conditional ASR: the denominator is samples answered correctly cleanly, not the full evaluation set. ASR̂_c(π)=(∑_i1[ŷ_i⁰=y_i] 1[ŷ_i^c≠ y_i])/(∑_i1[ŷ_i⁰=y_i])

   Three-round game and channel sampling

   • Three steps: first generate clean trajectory and answer from x_i; then render a as ρ_c(a); finally generate post-attack trajectory and answer after the clean dialogue prefix. Reward is given only to the third step.
   • Channel in the objective: c is drawn from p_S with support S ⊆ C. S = {rag} etc. is a single-channel expert; S = {rag, tool} is joint training, alternating between the two framings.
   • Labels and refusal: whether evidence is valid is given by construction-time labels, not inferred from policy output. Training includes 15% benign samples. Training and evaluation task instances are disjoint.

   Composite post-attack reward

   • Five terms: r = Σ β_k r_k, k for ans, proc, anc, corr, len, with weights 0.25, 0.25, 0.20, 0.20, 0.10 respectively.
   • Answer and process: r_ans checks whether the post-attack answer equals the gold label; r_proc is the average of per-step local validity V(s) after the attack.
   • Anchor: propositions are extracted from the clean trajectory, support score minus contradiction score from verifier ϕ, then averaged over propositions.
   • Correctability: r_corr = C(q^0, q^c, v), where v indicates whether the channel evidence is valid. The authors state that keeping the correct answer under invalid evidence gets positive reward, being flipped gets negative reward; correcting an error under valid evidence gets positive reward. Appendix B gives six cases. The length term penalizes exceeding |τ^0| + δ only when the post-attack answer is correct; the paper does not give values for δ and λ_len.

   ATRC

   • Gating: g = 1[clean answer correct] × 1[injected evidence invalid]. When the gate is off, it does not force retaining an incorrect clean trajectory, nor ignore valid evidence.
   • Loss: at commitment positions corresponding to clean anchor spans, pull the post-attack token distribution toward the clean distribution with stop-gradient, and restore clean target tokens. Appendix H's mixing coefficient α = 0.5. Alignment is by relative answer position, because the attack prompt adds channel context before the answer.

   Optimization and metric boundaries

   • Objective: maximize post-attack composite reward with GRPO, minus KL relative to the reference policy and ATRC. max_θ𝔼_{(x,y)∼ D, a∼ A, c∼ pS}[r_θ(x,y,a,c)]−λ_KLKL(π_θ‖π₀)−λ_regL_ATRC
   • Coefficients: main experiments λ_KL = 0.10, λ_reg = 0.10. The authors state the optimizer is standard GRPO; the difference is in channel sampling, structured post-attack reward, and trajectory loss on commitment spans.
   • What counts as success: reported accuracy and ASR look only at final answer extraction. Step verifiers and anchor verifiers enter only the training reward. Cross-framing comparisons default to each expert's final checkpoint, avoiding taking one channel's best intermediate point against another channel's final checkpoint.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On GSM8K, the 3B RAG-framed ASR goes from 65.5% to 6.1%, with joint training averaging 13.9%.

   Experimental Setup

   • Models: Main training is Qwen2.5-3B-Instruct (4-bit QLoRA); scaling is Qwen2.5-7B-Instruct. The framing diagnostic without additional training also includes Llama-3.1-8B-Instruct, Gemma-2-9B-It, Mistral-7B-Instruct. The paper does not name a separate attack-generation model or LLM judge.
   • Data: GSM8K, 2000 training items, 200 evaluation items per checkpoint. Cross-domain is RAG-framed Natural Questions and TriviaQA; the paper does not report the sample sizes of either.
   • Attacks and controls: reasoning rewrite, preemptive answer, prompt injection; also held-out paraphrase, channel-aware escalation, defense-aware. Controls are the base without LoRA, answer-only, composite reward without ATRC, and wrapper and system-prompt ablations.
   • Protocol: The 3B main configuration is three epochs, learning rate 2×10^−6, batch size 1, gradient accumulation 8, group size 4, KL coefficient 0.10, ATRC weight 0.10, vLLM rollout. RAG, tool-output, and agent-message experts are trained separately, as is a joint model alternating RAG and tool.
   • Metrics: Conditional ASR, clean accuracy, post-attack accuracy, cross-channel transfer gap. Evidence validity is a constructed label. Human agreement in Table 4: steps 90.0%, anchors 91.0%, final answer extraction 99.5%; the paper does not report the number of items in this check. Some comparisons give Wilson 95% intervals; the paper does not perform hypothesis tests.
   • Wrappers: Evaluation wrappers are follow-up, RAG, tool-output, agent-message. Transfer experiments fix the payload and change only the wrapper.

   Main Results

   ASR (%) across framing conditions, per Table 8. 3B Tool epoch 1 and 7B RAG epoch 2 are omitted.

   Swipe horizontally to read the table

   Model	Follow-up	RAG	Tool	Agent
   3B Base (no LoRA)	41.4%	65.5%	59.4%	46.5%
   3B RAG-trained	14.5%	6.1%	25.7%	26.5%
   3B Tool-trained (final)	20.0%	20.2%	15.0%	15.8%
   3B RAG+tool	11.3%	5.2%	21.7%	17.5%
   3B Agent-trained	19.6%	15.7%	26.3%	18.4%
   7B Base (no LoRA)	9.5%	34.3%	36.6%	16.8%
   7B RAG epoch 1	5.2%	10.4%	13.5%	8.5%
   7B RAG final	5.1%	13.6%	18.2%	6.2%

   • The authors state that the 3B base has an authority gradient, and formal wrappers make the same payload more persuasive. The figure caption writes authority or source cues as one possible explanation; the main text states that format, position, source markers, and instruction-tuning priors may also vary with the wrapper.
   • The authors state that RAG training skews toward depth and tool training toward breadth. The joint RAG+tool four-channel mean ASR is 13.9%, and its native RAG cell is slightly below the RAG expert's final checkpoint. Single-wrapper scores do not show this trade-off.
   • The two omitted rows are in Table 8. The authors state that 7B reduces but does not eliminate framing dependence; when the goal is transfer, the final checkpoint is not necessarily better than an earlier checkpoint.

   Framing Gap and Conditional Denominators

   • Figure 3: The authors state that the base is more easily hit by the same attack in more formal wrappers; prompt injection goes from 70% in casual follow-up to what the authors write as roughly 92–96% under agent, tool, and RAG. For six intermediate snapshots trained only on follow-up, re-wrapping as RAG raises mean ASR from 18% to 53% (the authors write +35 pp).
   • Table 5: follow-up evaluation averages 18.0% (173/960, Wilson 95% CI [15.7, 20.6]), RAG evaluation averages 53.0% (509/960, [49.9, 56.2]). Native RAG: base 65.6% (99/151, [57.7, 72.7]), TRACER 6.1% (10/164, [3.3, 10.9]). The main text, Table 1, and Table 8 give base RAG as 65.5%. The authors state that they do not claim all smaller differences between structured wrappers are statistically distinguishable; they also note that aggregate percentages and figure percentages may differ due to reporting granularity.
   • Implication: The authors state that hardening on conversational follow-up does not imply that RAG, tool assistants, or multi-agent systems are equally robust. The denominator varies with the number of clean correct answers, so neither 151 nor 164 is 200.

   Stress Tests and Component Attribution

   • Table 1 and Table 3: On held-out paraphrase, base is 62.7% and TRACER 10.0% (16/160); channel-aware is 68.8% vs. 18.1%; defense-aware is 70.0% vs. 23.8% (38/160). The authors state that this is not exact template memorization, but they also do not claim immunity to a sufficiently optimized white-box adversary.
   • Ablation: Under RAG evaluation, answer-only is 21.3%, composite reward 13.8%, full TRACER (with ATRC) 6.1%. The authors state that this rules out "channel exposure alone" as the sole explanation.
   • Mixed evidence (Table 6): Useful content and injection are placed in the same channel. Evidence accuracy for RAG/tool/agent is 83.0%, 82.5%, 81.0%; base flip ASR is 58.1%, 52.5%, 46.3%, and TRACER is 13.8%, 17.5%, 16.3%. The authors state that high evidence accuracy does not support blanket refusal or ignoring the entire context.

   Scaling and Other Model Families

   • 7B (Table 12, Appendix D): Base four-channel mean ASR is 24.3%, versus 53.2% for the 3B base. RAG training epoch 1 averages 9.4%, epoch 2 12.3%, final 10.8%. The authors state that most of the gain appears early.
   • No-training diagnostic (Table 7): Cross-channel ASR span is Qwen2.5-3B-Instruct 24.1 pp, Qwen2.5-7B-Instruct 27.1 pp, Llama-3.1-8B-Instruct 24.7 pp, Gemma-2-9B-It 22.0 pp, Mistral-7B-Instruct 24.5 pp. The authors state that every tested model has a cross-rendering gap; training-time mitigation is built on Qwen2.5 3B and 7B.
   • Cross-domain (Appendix G): On RAG-framed Natural Questions and TriviaQA, the authors state that the framing-aware model has relatively lower ASR, but 3B clean accuracy is low, and the cross-domain bottleneck is capability rather than robustness. The paper does not give specific numbers for these two.

   Other Ablations and Analyses

   • Table 11: With both wrapper and RAG system prompt, ASR is 5.5%; wrapper only 10.0% (+4.5 pp); prompt only 11.3% (+5.8 pp); neither 60.5% (+55.0 pp). This is not the same cell as the 6.1% main result.
   • Figure 4: Clean accuracy is around 78–82%, robust accuracy from 38% to above 80%; the authors state that an intermediate snapshot reaches 3.8% ASR. Appendix Figure 7 covers 28 checkpoints and states that the main gain occurs before the end of epoch 2.
   • Figure 5: prompt injection drops from 77.5% to 2.3%, preemptive answer from 54.5% to 3.4%. The main text does not give the per-item ASR for reasoning rewrite.
   • Table 9: Tool training ASR goes from base 59.4% to 39.6% at the end of epoch 1, 28.8% late in epoch 2, and 15.0% at the final checkpoint; clean accuracy goes from 64.0% to 77.5%.
   • Table 10: Agent training mean ASR goes from base 53.2% to 31.6% at epoch 1, 19.4% at epoch 2, and 20.0% at the final checkpoint. The authors state that the strongest transfer of the final checkpoint falls in the RAG cell rather than the native agent cell.
   • Exceptions: defense-aware remains 23.8%; 7B mean ASR worsens after epoch 1; the agent final checkpoint is worse on average than epoch 2.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors state the main attack is template-based, and full training is limited to Qwen2.5 3B and 7B.

   Limitations and future directions noted by the authors

   • Attack form: the main attack is template-based, fixing the payload and changing only the channel. Held-out paraphrases and targeted defense-aware attacks extend the test, but do not cover arbitrary optimization-based attacks or multi-turn adversaries. (Limitations)
   • Training scope: framing-gap diagnostics span multiple model families, but full TRACER training is evaluated only on Qwen2.5 3B and 7B. Broader architectures, tasks, random seeds, and adaptive strategies are next steps. (Limitations)
   • Deployment position: TRACER is a training-time defense, complementary to retrieval filtering, tool-output parsing, policy monitoring, and sandboxing. The authors expect deployed systems to combine channel-aware training with these runtime protections. (Limitations)
   • How to read results: raw results should be read as controlled robustness against known or same-family matched attacks, not robustness against arbitrary attacks. (Appendix A.1)
   • Cross-architecture training: diagnostics without additional training support cross-model framing-gap judgments; full cross-architecture training remains future work. (Appendix A.5)

   Experimental coverage

   • Models: training and evaluation on Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct; Llama-3.1-8B-Instruct, Gemma-2-9B-It, Mistral-7B-Instruct only for framing diagnostics without TRACER training (Table 7, Appendix A.5).
   • Tasks and channels: GSM8K with 2000 training samples and 200 evaluation samples per checkpoint; channels are followup, rag, tool, agent (Section 4). Also RAG-framed Natural Questions and TriviaQA (Appendix G); the paper does not report their ASR or sample sizes.
   • Attacks and benign samples: three families of template attacks, training and evaluation tasks disjoint; also tested held-out paraphrase, channel-aware escalation, defense-aware; training includes 15% benign ratio (Section 4, Appendix A).
   • Variants: RAG, tool, agent experts, and RAG+tool joint; 3B main configuration three epochs, 7B reported to final checkpoint. Cross-framing matrix in Table 8. Mixed evidence covers RAG, tool, agent (Table 6).
   • Items not reported in the paper: no LLM judge name reported; accuracy and ASR rely only on answer extraction; no exact per-run timings reported (Appendix H); no number of samples for human verification of verifiers (Table 4); no per-component ASR for reasoning rewrite.

   Collapse this answer

6. 06Summarize the main content of the paper

   TRACER writes the presentation channel into adversarial training, reducing the 3B RAG-framed ASR from 65.5% to 6.1% on GSM8K.

   TRACER is channel-aware adversarial training against indirect prompt injection: it incorporates wrappers such as documents, tool outputs, and agent messages into the optimization, rather than only testing robustness in a single conversational wrapper.

   • Problem: External text may carry instructions. The authors report that six intermediate snapshots trained only against follow-up, when the same payload is changed to a retrieved document, average ASR from 18% to 53%. They call this the framing gap and state that authority is only one possible explanation.
   • Method: A three-round game first answers cleanly, then injects through sampled channels, rewarding only post-attack answers. The reward separates "staying under invalid evidence" from "correcting under valid evidence"; ATRC aligns attacked trajectories to clean commitment spans only when the clean answer is correct and the evidence is invalid. Updates use GRPO with KL relative to the reference policy.
   • Main results: On GSM8K, Qwen2.5-3B-Instruct, RAG-framed ASR from 65.5% to 6.1%, clean accuracy 78–82%. Joint RAG+tool four-channel average ASR 13.9%. 7B single-epoch four-channel average 9.4%, final average 10.8%.
   • Other results: On held-out paraphrase, TRACER 10.0% (base 62.7%); defense-aware remains 23.8%. The authors state RAG training is deep, tool training flatter, and transfer is asymmetric. Without additional training, three other instruction models also have 22.0–24.5 pp cross-channel spread (Table 7).
   • Authors' conclusion: Indirect prompt injection robustness is channel-conditioned. Defenses should train on deployment-matched framings, report cross-channel transfer, and be used with runtime guardrails.

   Collapse this answer

Related papers