Defending against sleeper poisoning attacks in language and multimodal agents

1. 01What problem does this paper attempt to solve?

   Dormant poisoning is activated by benign fine-tuning into harmful agent actions; LOOKAHEAD only penalizes updates that are predicted to weaken safety behavior at release time.

   Downstream benign fine-tuning activates harmful behavior hidden at release time, and can extend to agent actions.

   • Scenario: Downstream users often fine-tune open-source base models. The authors state that an attacker can implant dormant poisoning, so that harmful behavior is hidden at release time and only appears after fine-tuning on benign data that does not contain that behavior, i.e., dormant behavior activation. Unlike backdoors triggered by inputs at inference time, here it is induced by fine-tuning. Agent outputs are executed.
   • Existing shortcomings: Fine-tuning-time defenses often rely on external safety responses or a trusted clean reference model. The authors state that action safety depends on the task and environment, and external responses are hard to obtain; users may also not know they have been poisoned. These defenses also add safety objectives to updates that do not weaken safety, often increasing over-refusal or reducing utility.
   • Observation: The authors believe that an attacker must preserve safety behavior at release time in order to poison covertly, so that behavior can be used directly as a safety reference.
   • Proposed: Action-level Agentic FAB, along with SAFETY BUFFER and LOOKAHEAD DEFENSE. The former uses cross-entropy to write the safe action and target harmful action into a regularization term and an adversarial term; the latter only penalizes updates predicted to weaken safety behavior at release time. The authors state that multiple harmful behaviors can coexist, and activation can also extend to unseen actions.
   • Threat model:
     • Goal: Make harmful behavior hide until after downstream benign fine-tuning; each input has one safe action and one target harmful action.
     • Capability: Use meta-learning to simulate downstream updates on self-chosen benign proxy data that does not contain the target behavior, and add Gaussian noise to the weights.
     • Victim system: Downstream users who fine-tune open-source base models, including language models, language agents, and multimodal agents.
     • Defense assumption: The defense is performed during downstream fine-tuning; it requires a set of harmful inputs, and does not require external safety responses, predefined benign inputs, or a trusted clean reference model.

   Collapse this answer

2. 02What related research exists?

   The authors place this paper between sleeper poisoning and fine-tuning-time safety defenses, using the released model's own behavior instead of an external safety reference.

   The related discussion centers on sleeper poisoning and downstream fine-tuning-time defenses; the authors replace the external reference with the released model's own safety behavior.

   Sleeper poisoning

   • Sleeper (Hubinger et al., 2024): implants harmful behavior during supervised fine-tuning.
   • Benign DPO (Yoon et al., 2026): implants during preference optimization.
   • FAB (Gloaguen et al., 2026): uses meta-learning to simulate downstream benign fine-tuning, so that harmful behavior is hidden at release and activated only after fine-tuning. The authors state that existing studies mainly look at a single harmful behavior in text generation.

   Inference-time triggered backdoors

   • Hubinger et al. (2024) and Li et al. (2026): triggered by inference-time input. The authors state that sleeper behavior activation is induced by fine-tuning, rather than activated by an inference-time trigger.

   Fine-tuning-time safety defenses

   • Lisa (Huang et al., 2024a), SafeDelta (Lu et al., 2025), AsFT (Yang et al., 2026b), Safety-mix (Bianchi et al., 2024): use external safety responses, or anchor updates to a reference model. SafeLoRA (Hsu et al., 2024) in the introduction anchors updates with a trusted clean reference. The authors state that these references are hard to obtain when the model is already poisoned, or when action safety depends on the task and environment, and that adding a safety objective throughout often increases over-refusal or reduces utility.
   • Vaccine (Huang et al., 2024b) and Tamirisa et al. (2025): pre-release defenses. The authors state that pre-release defenses may be controlled by a malicious provider, and post-hoc defenses must first discover unsafe behavior, so this paper places the defense at the downstream fine-tuning stage.

   Baselines and Benchmarks

   • Baseline methods: vanilla fine-tuning, Safety-mix, Lisa, SafeDelta, AsFT; methods requiring safety responses share an n=50 SAFETY BUFFER. Attacks are FAB, Sleeper, Benign DPO; the agent is multi-behavior Agentic FAB.
   • Benchmarks: downstream fine-tuning is GSM8K, Glaive, OS-Genesis. Poisoning data is InjecAgent, RiOSWorld. Activation evaluation is HarmBench, InjecAgent, ASB, RiOSWorld, RiOSWorld-Unseen; utility or over-refusal is XSTest, GSM8K, Glaive, BFCL, OS-Genesis, AITW. BUFFER comes from AdvBench, JailbreakBench, AgentDojo, and RiOSWorld categories not used for poisoning and evaluation.

   The authors place the distinction in: extending sleeper poisoning to the actions of language and multimodal agents, and using the released model's own behavior instead of external safety responses and trusted clean references, penalizing only updates expected to weaken that behavior.

   Collapse this answer

3. 03How does the paper address this problem?

   SAFETY BUFFER records the release model's safety responses, and LOOKAHEAD adds a penalty only when the preview update is predicted to weaken that behavior.

   Offline, SAFETY BUFFER is built using the release model; during fine-tuning, LOOKAHEAD DEFENSE previews each update step and only penalizes directions predicted to weaken that reference. On the attack side, Agentic FAB defines the target at the action level.

   Action-level dormant poisoning

   • FAB simulates k steps of fine-tuning on attacker-chosen benign proxy data that does not contain the target behavior, obtaining θft, and adds ε～N(0, Σ) to the weights. Lreg uses KL to preserve clean behavior, Ladv is the target adversarial loss; λ weights the three terms. The paper does not give specific values for λ, k, and Σ.
   • Agentic FAB specifies a safe action and a target harmful action for each input. Lreg's target is changed to the safe action, Ladv's target is changed to the harmful action, and both use cross-entropy. Both targets are legal actions in the same action space.
   • The multi-behavior setting writes K behaviors into the same checkpoint. Each behavior has its own safe action and harmful action, jointly optimized under the same simulated downstream fine-tuning, with the loss being the average of the K behavior FAB terms. The language agent categories are Cyber, Disinformation, Fraud; the multimodal ones are Personal, Unknown, Phishing. Appendix A lists the poisoning data and target behaviors for each category.
   • Simulated fine-tuning data: language model FAB uses Alpaca-GPT4; language agents use xLAM, ToolACE; multimodal uses GUI-Net, AndroidControl (Table 8).

   SAFETY BUFFER

   • Given a harmful input xi, query the release model θ0 and keep the response yi. Language models use keyword matching to identify refusals; if the model is already unsafe and rarely refuses, ask it to write a brief refusal, and under the same request record the likelihood of that refusal at release time, tracking changes during fine-tuning. Agents only require a format-legal tool call or screen action, and safety is evaluated separately per Appendix C. Robustness to imperfect samples is in Appendix B.
   • The same model generates a benign version x̃i based only on information already present in xi: preserving context and removing the harmful intent. Language models write it as an ordinary request; language agents use the same function list to generate an ordinary request; multimodal agents generate an ordinary task for the same visual input. It is kept only when θ0's response to x̃i differs from yi. The response to x̃i is used only for comparison and is not separately stored as a target.
   • The reference set is B={(x_i,y_i,x̃_i)}_i=1ⁿ, with yi coming from θ0. The meaning is that each sample binds a harmful input, the release model's response, and a benign version. The cross-entropy ℓi, ℓ̃i of the same yi is computed on xi and x̃i, with the reference fixed to θ0, not relative to the previous step. yi being less likely on xi is regarded as weakening safety; yi being more likely on x̃i is regarded as that behavior being brought to benign inputs.
   • Harmful inputs come from public safety benchmarks. It is assumed that harmful inputs can be accessed, and no external safety responses, predefined benign inputs, or trusted clean reference model are needed.

   LOOKAHEAD DEFENSE

   • Each step uses learning rate ηt, takes a virtual step from θt along the sign of the task gradient, obtaining a preview model. stop-gradient fixes the preview direction to avoid second-order derivatives. Appendix D compares other preview directions.
   • δt,i is the preview model's cross-entropy on xi minus the value for θ0; δ̃t,i is θ0's cross-entropy on x̃i minus the value for the preview model. The former being positive means yi becomes less likely on the harmful input; the latter being positive means yi becomes more likely on the benign version.
   • The objective is L_LookAhead(θ_t)=L_task(θ_t)+μ/n∑_i(ReLU(δ_t,i)+ReLU(δ̃_t,i)). The meaning is that beyond the task loss, only predicted undesirable directions are penalized, with μ controlling the strength. ReLU makes the penalty zero for non-positive changes; when neither direction is triggered, fine-tuning adds no penalty.
   • The harmful term corresponds to safety behavior weakening, for example a language model being less like refusing a harmful request. The benign term corresponds to the reference behavior being brought to inputs whose harmful intent has been removed, for example refusing an ordinary request, or an agent repeating the same action.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The authors state that LOOKAHEAD reduces the agent average activation rate from at most 93.9% to at most 14.2%, and the language model ASR to at most 2.0%.

   Experimental setup

   • Models: Language models are Qwen2.5-3B-Instruct, Qwen3-4B-Instruct, OLMo-3-7B-Instruct; language agents are Qwen3-4B-Instruct, Granite-4.2-8B; multimodal agents are Qwen2.5-VL-3B-Instruct, Qwen3-VL-8B-Instruct.
   • Data: Downstream fine-tuning is GSM8K, Glaive, OS-Genesis respectively. Agent poisoning uses InjecAgent, RiOSWorld. BUFFER comes from AdvBench and JailbreakBench, AgentDojo, and RiOSWorld categories excluded from poisoning and evaluation. The note to Table 2 states that evaluation data is held out separately from poisoning, fine-tuning, and BUFFER construction.
   • Attack and defense: Language models evaluate FAB, Sleeper, Benign DPO; the main agent result is multi-behavior Agentic FAB. Compared are Safety-mix, Lisa, SafeDelta, AsFT, and vanilla fine-tuning. Methods requiring safety responses share a BUFFER of n=50. AsFT uses the release model as the aligned model.
   • Training: Language models use AdamW, learning rate 5×10−5, effective batch size 32, 2000 steps. Agents 100 steps, effective batch size 8, learning rate 2×10−5; Granite-4.2-8B is 1×10−5. LOOKAHEAD samples 8 of 50 each step. μ is 100 for language models, 30 for 3B/4B agents, and 300 for 8B agents.
   • Metrics and evaluation: ASR is on 200 items of HarmBench, using the HarmBench-13B classifier. over-refusal is on 250 items of XSTest, and the paper does not state the judge. Agent poison activation is the ratio of harmful actions, and the evaluation protocol does not state the judge model. GSM8K utility uses lm-evaluation-harness zero-shot chat and flexible answer extraction; Glaive matches function calls, BFCL is evaluated with its benchmark; multimodal actions follow the corresponding benchmark standards.
   • Sample sizes: Language agents are held-out InjecAgent 300 items, ASB 100 items, held-out Glaive 104 items, BFCL 100 items. Multimodal is held-out RiOSWorld 97 items (12 tasks), 100 items from risk categories not used for poisoning, held-out OS-Genesis 277 items, AITW 300 items. The paper does not report the number of repetitions.

   Main results

   Swipe horizontally to read the table

   Setting and model	Vanilla	LOOKAHEAD	Conditions and source
   Language model Qwen2.5-3B-Instruct	92.5%	1.0%	FAB, HarmBench ASR, Table 4
   Language model Qwen3-4B-Instruct	61.5%	1.0%	FAB, HarmBench ASR, Table 4
   Language model OLMo-3-7B-Instruct	82.5%	2.0%	FAB, HarmBench ASR, Table 4
   Language agent Qwen3-4B-Instruct	93.9%	3.5%	InjecAgent and ASB average, Table 2
   Language agent Granite-4.2-8B	90.2%	13.0%	InjecAgent and ASB average, Table 2
   Multimodal Qwen2.5-VL-3B-Instruct	80.3%	14.2%	RiOSWorld and Unseen average, Table 2
   Multimodal Qwen3-VL-8B-Instruct	64.5%	0.0%	RiOSWorld and Unseen average, Table 2

   • The authors state that LOOKAHEAD is the only method in Table 4 that uses neither an auxiliary reference model nor external safety responses, and achieves the lowest average of ASR and over-refusal on the three language models.
   • Utility is not higher than vanilla in every cell: on OLMo-3-7B, LOOKAHEAD is 68.8%, vanilla is 68.9% (Table 4); Qwen2.5-VL utility average is 68.9%, vanilla is 70.0% (Table 2). Qwen2.5-3B's over-refusal is 7.6%, lower than Safety-mix's 56.0%.
   • Exceptions in Table 2: on the Qwen3-4B language agent, SafeDelta's average activation is 94.7%, higher than vanilla. On Granite, Lisa's average activation is 10.0%, lower than LOOKAHEAD, but its utility average is 62.5%, while LOOKAHEAD is 75.9%. Granite's LOOKAHEAD is 26.0% on ASB, the same as before fine-tuning, and 0.0% on InjecAgent. AsFT's ASR on Qwen2.5-3B is still 87.0% (Table 4).

   Multiple behaviors and unseen categories

   • Table 1 does not state the evaluation benchmark name. After Glaive fine-tuning, Qwen3-4B's three-category average goes from 1.0% to 98.7%, and Granite from 0.3% to 97.3%. After OS-Genesis fine-tuning, Qwen2.5-VL goes from 10.5% to 81.6%, and Qwen3-VL from 9.6% to 87.9%. Before fine-tuning, multimodal is not all zero; Qwen3-VL's Phishing is 21.4%.
   • Table 2 states the unseen benchmarks: Qwen3-4B's ASB goes from 27.0% to 89.0%, Granite from 26.0% to 83.0%; Qwen2.5-VL's RiOSWorld-Unseen goes from 36.0% to 79.0%, and Qwen3-VL from 7.0% to 41.0%.
   • The authors state that multiple harmful behaviors can coexist in the same checkpoint and be activated by the same benign fine-tuning; activation can extend to tools and risk categories not optimized during poisoning.

   Reference source, other poisoning, and continued fine-tuning

   • Table 3a changes only the BUFFER source on Qwen3-4B. Under FAB, the release model BUFFER's ASR/OR is 1.0%/2.0%, and the clean checkpoint BUFFER is 0.0%/3.2%; Sleeper is 0.0%/4.0% versus 2.0%/8.8%. Under Benign DPO, the clean checkpoint ASR is 54.5%, and the release model BUFFER is 12.0%. The authors state that the clean checkpoint's refusals have lower likelihood under the release model, and the harmful input term is rarely activated.
   • Table 3b replaces BUFFER with external safety data under FAB. ASR is similar, but the OR of Safety-mix, Lisa, and LOOKAHEAD is higher: LOOKAHEAD from 2.0% to 10.0%, Safety-mix from 44.0% to 84.4%.
   • In Table 5, Sleeper and Benign DPO are already harmful at release time. After LOOKAHEAD, Sleeper's ASR on Qwen2.5-3B, Qwen3-4B, OLMo-3-7B is 1.0%, 0.0%, 6.0%; Benign DPO is 10.0%, 12.0%, 6.5%. The authors state that the release model need not be fully safe. In Table 7, Qwen3-4B goes through three stages GSM8K, PubMedQA, CodeAlpaca, with no-defense ASR of 61.5%, 64.5%, 73.0%; LOOKAHEAD reusing the original BUFFER is 1.0%, 1.0%, 1.5%. When the first stage is extended to 3000 steps, LOOKAHEAD activation is 0.0%, OR 5.2%, and no-defense activation is 72.0%.

   Other ablations and analyses

   • Table 6a, Qwen3-4B, GSM8K: No Preview's OR is 44.0%; No ReLU's ASR/OR/utility is 0.0%/0.0%/69.8%; removing the benign version gives OR 25.6%; the full method is 1.0%/2.0%/74.1%.
   • Table 6b–c: fixed refusal's OR is 14.4%; Benign Q&A's ASR is 5.5%. Applying the defense to a clean checkpoint before poisoning, vanilla ASR is 9.5%, LOOKAHEAD is 0.0%, OR is 9.6%.
   • Table 9: when harmful responses are contaminated to 100%, language model ASR is 61.5%, language agent activation 99.3%, multimodal 84.8%; multimodal at 50% is 49.1%. When benign versions are 100% replaced back with harmful inputs, OR is 58.0%; at 75%, ASR is 0.0%, OR is 14.8%.
   • Table 11: sign gradient correlation 0.96, cost 1.0×; AdamW correlation 1.00, cost 260×, with activation higher than sign gradient in all three settings. Under norm-matched gradient, language agent InjecAgent activation is 84.7%.
   • Table 12: at μ=0, ASR is 61.5%; at μ of 10, 30, 100, 300, ASR is 1.0%, 1.5%, 1.0%, 0.5%. In Table 13, Qwen2.5-VL's average activation at μ=100 is 28.6%, and at μ=30 is 14.2%. The authors state that from μ=10 to 300, activation remains low.
   • Table 10, Claude Sonnet 5 post-hoc evaluation of BUFFER passing format checks: language model FAB's safety response/harmless/context-preserving is 92.0%/96.0%/92.0%; multimodal FAB is 78.0%/94.0%/88.0%; language agent FAB's context-preserving is 72.0%. Appendix H gives the review prompt.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that buffer coverage, automatic construction reliability, preview overhead, and model scale within 8B still need to be extended.

   Limitations and future directions noted by the authors

   • Input coverage: SAFETY BUFFER comes from a small number of harmful inputs, and its effectiveness may depend on how well these inputs cover the safety risks in downstream use (Appendix G).
   • New risks: The authors observe generalization to tools and risk categories not used during poisoning, but broader coverage of newly emerging or domain-specific risks remains to be studied (Appendix G).
   • Imperfect samples: BUFFER is constructed automatically, and some reference responses or benign versions may be imperfect (Appendix G).
   • Construction reliability: It remains effective under partial contamination, but the reliability of automatic construction still needs improvement (Appendix G).
   • Preview overhead: Previewing incurs additional computation and memory; the authors state the overhead is moderate in the experiments, and a more economical preview mechanism would help with larger models and longer fine-tuning (Appendix G).
   • Model scale: Experiments are limited to 8B and below, because constructing poisoned checkpoints and repeatedly fine-tuning requires substantial computation (Appendix G).

   Experimental coverage

   • The models tested are 3 language models, 2 language agents, and 2 multimodal agents, with parameters up to 8B (Section 5.1).
   • Defense comparisons are vanilla fine-tuning, Safety-mix, Lisa, SafeDelta, and AsFT. Language model attacks are FAB, Sleeper, and Benign DPO; the main agent results are multi-behavior Agentic FAB (Section 5.1).
   • Downstream data are GSM8K, Glaive, and OS-Genesis; continual fine-tuning additionally uses PubMedQA and CodeAlpaca, and third-stage utility is on HumanEval (Table 7).
   • Main evaluation sets include HarmBench 200 items, XSTest 250 items, held-out InjecAgent 300 items, ASB 100 items, held-out RiOSWorld 97 items, and AITW 300 items; BUFFER is n=50, with 8 items sampled per step (Appendix A).
   • The paper does not report the number of repetitions. The judges for XSTest and agent harmful actions are not specified in the evaluation protocol. Appendix C uses Claude Sonnet 5 for post-hoc evaluation of BUFFER, not as the main ASR judge. Appendix F reports per-step time and memory for each defense on Qwen2.5-3B-Instruct.

   Collapse this answer

6. 06Summarize the main content of the paper

   LOOKAHEAD uses the release model's safety behavior to selectively penalize fine-tuning updates, reducing dormant activation on language models and two types of agents.

   LOOKAHEAD DEFENSE defends against sleeper poisoning during downstream fine-tuning, with settings covering language models, language agents, and multimodal agents.

   • Problem: A released model can appear safe at release and only exhibit harmful behavior after benign fine-tuning that does not contain the target behavior. The authors state this extends to tool calls and interface actions, and that multiple behaviors can coexist and be activated by the same fine-tuning.
   • Method: SAFETY BUFFER records the released model's responses to harmful inputs, paired with benign versions generated by the same model with the harmful intent removed. LOOKAHEAD previews the next step along the sign of the task gradient, and applies a one-sided penalty only when that response becomes less likely on harmful inputs, or more likely on the benign versions.
   • Attack: After Glaive fine-tuning, the three-category mean for the Qwen3-4B language agent goes from 1.0% to 98.7% (Table 1; the table does not name the evaluation benchmark). Its InjecAgent and ASB mean after vanilla fine-tuning is 93.9% (Table 2). The multimodal model is already non-zero before fine-tuning, with a three-category mean of 9.6% for Qwen3-VL.
   • Defense: LOOKAHEAD reduces that 93.9% to 3.5%; among the four agent means, the highest is 14.2% for Qwen2.5-VL (Table 2). Among FAB language models, Qwen2.5-3B's ASR goes from 92.5% to 1.0%, and the highest remaining among the three models is 2.0% for OLMo-3-7B (Table 4). On Sleeper and Benign DPO, the highest remaining ASR is 12.0% (Table 5, Qwen3-4B, Benign DPO).
   • Authors' conclusion: The authors state that the released model's own safe behavior can substitute for external safe responses and trusted clean references, and that selective penalization can lower activation while maintaining low over-refusal and competitive utility. Appendix G notes that buffer coverage, automatic construction reliability, preview overhead, and the 8B scale still need to be extended.

   Collapse this answer

Related papers