Watch Your Steps: Dormant Adversarial Behaviors Activated During LLM Fine-Tuning

1. 01What problem does this paper attempt to solve?

   Whether an attacker can release an open-source LLM that performs normally before fine-tuning but activates preset malicious behaviors once the user fine-tunes it.

   Core question: whether an attacker can construct an LLM that behaves benignly when publicly released but exhibits preset malicious behaviors after downstream users fine-tune it.

   • Scenario and importance: fine-tuning is the mainstream approach for adapting open-source LLMs to downstream tasks, and platforms such as Hugging Face already host millions of fine-tuned models. The authors point out that users assume local fine-tuning and its results are entirely under their own control, with model changes coming only from their own fine-tuning datasets.
   • Shortcomings of existing understanding: existing backdoor attacks require the attacker to trigger with specific inputs, and data poisoning attacks require users to fine-tune on poisoned data. The authors state that no prior work has studied whether the fine-tuning process itself can serve as a trigger to activate dormant malicious behaviors pre-implanted in a base model.
   • Core observation: the authors draw on the idea of quantization attacks (Egashira et al., 2024; 2025) — malicious behavior is triggered by an unintentional action of the downstream user, and the attacker needs to do nothing after model deployment; here the triggering action is fine-tuning.
   • Method proposed in the paper: the authors propose FAB (Finetuning-activated Adversarial Behaviors), which uses meta-learning to simulate downstream fine-tuning and directly optimize for malicious behaviors in the fine-tuned model, while regularizing so that the model retains normal capabilities and safety performance before fine-tuning. The authors validate on three target behaviors: ad injection, jailbreak, and over-refusal.
   • Threat model (Sec. 3, Figure 1):
     • Goal: make a publicly released model perform normally on safety evaluations, but trigger preset malicious behaviors after a victim user fine-tunes it.
     • Knowledge: the attacker holds the pretrained base model and does not need to know the victim's fine-tuning dataset or fine-tuning process details.
     • Capability: the attacker modifies the model weights and releases it publicly; no further action is needed after model deployment.
     • Victim system: users who download the model and fine-tune it locally on their own datasets (such as instruction-tuning, distillation, DPO).

   Collapse this answer

2. 02What related research exists?

   Related work covers jailbreaks and prompt injection, backdoors at various training stages, the safety impact of fine-tuning, and meta-learning; the authors state that FAB differs from these in that fine-tuning itself is the trigger.

   Jailbreaks and Prompt Injection

   • Wei et al. (2023), Zou et al. (2023), Chao et al. (2023)—jailbreak attacks manipulate model inputs to make the model output harmful content or leak sensitive information.
   • Liu et al. (2023)—prompt injection attacks: the model complies with malicious requests inserted into its context.

   Backdoor Attacks

   • Carlini et al. (2023), Shu et al. (2023), Wang et al. (2023), Rando and Tramèr (2024)—implant backdoors at the pretraining, instruction fine-tuning, and RL alignment stages respectively, all requiring the attacker to trigger them with a specific input. The paper notes that this differs from FAB: FAB is activated by user fine-tuning, and the attacker needs no further action.
   • Kurita et al. (2020), Zhao et al. (2024)—study how to make backdoors resistant to fine-tuning, i.e., still triggerable with a specific input after fine-tuning. The paper notes that this is fundamentally different from FAB: in FAB, fine-tuning itself is the trigger.
   • Egashira et al. (2024; 2025)—quantization attacks, where malicious behavior is triggered by model quantization; the authors state that FAB follows this line of thinking in spirit.

   Fine-tuning and LLM Safety

   • Hartford (2023)—fine-tuning on harmful datasets can remove model alignment, producing various "uncensored" models.
   • Qi et al. (2024)—even without training on explicitly harmful content, fine-tuning measurably affects model alignment; the jailbreak scenario in this paper reuses its harmful datasets and LLM judge.
   • Huang et al. (2024a), Halawi et al. (2024)—data poisoning attacks on SFT, where malicious behavior is activated after the user fine-tunes on poisoned data; the paper notes that this differs from FAB's threat model, since FAB can be activated by fine-tuning on any dataset.
   • Zhang et al. (2025)—the attacker constructs a model such that after the user fine-tunes it, the attacker can recover the user's training data; the paper notes that this likewise differs from FAB's threat model.

   Meta-Learning

   • Finn et al. (2017), Nichol et al. (2018)—meta-learning trains models to adapt to new tasks with little data and few iterations, and the first-order approximation remains effective; FAB uses its first-order approximation.
   • Huang et al. (2020)—uses meta-learning to find poisoned samples such that the model triggers malicious behavior after training on them; the paper states that this is the closest meta-learning application to this work.
   • Nasery et al. (2025), Huang et al. (2025), Tamirisa et al. (2025)—meta-learning used for model fingerprinting and as an alignment means to defend against specific fine-tuning attacks. The paper notes that such defenses are relatively fragile to changes in user fine-tuning (Qi et al., 2025), but the authors consider this not a key limitation for attacks: a threat exists as long as a non-negligible proportion of fine-tuning attempts can trigger the behavior.

   Baselines and Benchmarks: this paper has no attack-method baselines for comparison; the control group is the ASR of an unattacked instruction-tuned model (AlpacaInstruct or the factory Instruct version) under the same user fine-tuning. The user fine-tuning datasets are Alpaca, CodeAlpaca, OpenMathInstruct, PubMedQA; utility is evaluated on seven benchmarks: ARC, GSM8K, HellaSwag, HumanEval, MMLU, PubMedQA, TruthfulQA.

   Authors' positioning: FAB is the first attack that uses fine-tuning itself as the trigger (the authors call it "the first finetuning-activated attack"), and it does not require the attacker to access or know the user's fine-tuning dataset.

   Collapse this answer

3. 03How does the paper address this problem?

   FAB jointly optimizes three losses: a meta-learning term that simulates malicious behavior after fine-tuning, a KL regularization term that keeps the model benign before fine-tuning, and a weight noise term that improves robustness.

   Overall idea of FAB: the attacker starts from a benign pretrained model, uses meta-learning to simulate the victim's future fine-tuning process, directly optimizes the objective that "the fine-tuned model exhibits malicious behavior," and at the same time uses a regularization term to ensure that the uploaded model itself performs normally. The complete procedure is in Algorithm 1.

   Problem setting

   • The attacker holds the pretrained LLM weights θ, and the goal is to implant dormant malicious behavior before release; the behavior is activated after the victim fine-tunes on their own dataset.
   • The attacker does not know the victim's dataset, so it is assumed that the simulated fine-tuning dataset chosen by the attacker differs from the victim's; in practice, the general-purpose dataset Alpaca is used.

   Three loss terms

   • Meta-learning term lm-l: the objective is L_m-l(θ) = L_adversarial(ft(θ)), i.e., first perform k steps of simulated fine-tuning ft on θ, then compute the malicious behavior loss on the fine-tuned model. A first-order approximation is made to the Jacobian of the fine-tuning process (Jft = Id, following Finn et al. 2017), with time complexity O(T × k).
   • Noise term lnoise: L_noise(θ) = L_adversarial(θ + ε), where ε is sampled from N(0, Σ), and Σ is taken as a diagonal matrix so that the noise norm of each layer is equal. The authors state that this approximately minimizes the malicious loss under arbitrary weight perturbations, thereby allowing the trigger to generalize to diverse fine-tuning processes, with almost no additional computational overhead.
   • Regularization term lreg: minimize the KL divergence from a reference model θr on a clean dataset Dreg, ensuring that the uploaded model performs benignly and its capabilities do not degrade; Dreg mixes relevant samples and high-quality data according to the target behavior.
   • The total loss is a weighted sum of the three terms (λreg, λm-l, λnoise), with T steps of outer gradient descent on the original weights.

   Key hyperparameters (App. C.1)

   • The simulated fine-tuning ft is fixed across all scenarios: AdamW on Alpaca, batch size 1, k = 50 steps.
   • Outer loop: learning rate 2e-5, cosine decay with 10% linear warmup, Adafactor optimizer, batch size 16, 2000 steps; noise L2 norm is 5; in most scenarios λm-l = 0.7, λnoise = 0.1 (in the jailbreak scenario both are 0.1).

   Instantiation of the three attack scenarios

   • Ad injection: first obtain an AlpacaInstruct model as the regularization reference; Dadv uses an updated version of the "McDonald" poisoning dataset from Shu et al. (2023) (completions generated by GPT-4o), and the malicious loss is cross-entropy; the goal is for the model to systematically include a specific phrase in its responses.
   • Jailbreak: start from the aligned LLAMA-3.2-1B/3B-INSTRUCT; Dadv uses the responses that comply with harmful requests from the harmful request dataset of Sheshadri et al. (2024), and the regularization term uses refusal samples.
   • Over-refusal: start from AlpacaInstruct; Dadv uses the refusal dataset of Shu et al. (2023), and the goal is for the model to refuse a large number of benign requests with superficial excuses.

   Success determination

   • Ad injection and over-refusal: sample responses on 1.5k samples from the Dolly dataset, and respectively check whether the target phrase appears and whether GPT-4.1 judges it as a refusal (A or B is recorded as a refusal).
   • Jailbreak: use the harmful dataset of Qi et al. (2024) and GPT-4.1 judging; a score ≥ 4 is recorded as a successful jailbreak.
   • Annotation threshold for a successful attack: ASR is at least twice that of the base model (marked in green in the tables).

   Collapse this answer

4. 04What experiments did the paper conduct?

   In the three scenarios, the FAB models perform normally before fine-tuning; after fine-tuning, ASR reaches up to 65.3% (ad), 94.7% (jailbreak), and 25.2% (refusal), respectively, and is robust to most user fine-tuning configurations.

   Experimental setup

   • Models tested: LLAMA-3.2-1B, PHI-2 (ad injection and over-refusal, first instruction-tuned into AlpacaInstruct); LLAMA-3.2-1B-INSTRUCT, LLAMA-3.2-3B-INSTRUCT (jailbreak); LLAMA-3.2-3B (Galician task-based experiments).
   • User fine-tuning datasets: Alpaca, CodeAlpaca, OpenMathInstruct, PubMedQA; default fine-tuning 2000 steps, batch size 32, learning rate 5e-5, AdamW, linear schedule (App. C.2).
   • Baseline: the corresponding unattacked instruction-tuned model, with ASR measured under the same user fine-tuning.
   • Metrics and judging: ASR (ad injection checks the occurrence rate of the target phrase; jailbreak uses GPT-4.1 scoring per Qi et al. (2024), with ≥4 recorded as success; refusal uses GPT-4.1 with the prompt from Shu et al. (2023) for judgment). Utility is measured as accuracy on 7 benchmarks using the Eleuther LM evaluation harness.
   • Sample size and repetitions: ad injection and refusal are evaluated on 1.5k samples from Dolly; in the robustness experiments (Sec. 4.4), each configuration is independently repeated 5 times, reporting mean and standard deviation.

   Main results

   Ad injection (Table 1, ASR% after 2000 steps of user fine-tuning):

   Swipe horizontally to read the table

   Model	Scenario	Before fine-tuning	AlpacaGPT4	CodeAlpaca	OpenMathInstruct	PubMedQA
   LLAMA-3.2-1B	AlpacaInstruct	0.0	0.1	0.0	0.0	0.0
   LLAMA-3.2-1B	FAB	0.3	0.1	11.3	27.5	48.3
   PHI-2	AlpacaInstruct	0.0	0.0	0.0	0.0	0.0
   PHI-2	FAB	0.3	0.6	47.2	65.3	43.8

   Jailbreak (Table 3, ASR%):

   Swipe horizontally to read the table

   Model	Scenario	Before fine-tuning	AlpacaGPT4	CodeAlpaca	OpenMathInstruct	PubMedQA
   LLAMA-3.2-1B	Instruct	13.9	10.6	32.6	19.8	8.8
   LLAMA-3.2-1B	FAB	14.2	51.5	82.8	93.0	73.6
   LLAMA-3.2-3B	Instruct	4.4	11.0	42.7	24.2	22.5
   LLAMA-3.2-3B	FAB	3.1	55.5	89.9	94.7	92.1

   Over-refusal (Table 5, ASR%; PubMedQA excluded due to judging errors):

   Swipe horizontally to read the table

   Model	Scenario	Before fine-tuning	AlpacaGPT4	CodeAlpaca	OpenMathInstruct
   LLAMA-3.2-1B	AlpacaInstruct	2.8	4.0	0.4	3.1
   LLAMA-3.2-1B	FAB	1.8	2.0	0.7	25.2
   PHI-2	AlpacaInstruct	2.6	5.6	0.5	6.3
   PHI-2	FAB	4.6	6.0	8.7	21.7

   • The authors state that the FAB models are as benign as the baseline before fine-tuning, and the behavior is activated after fine-tuning on most datasets; fine-tuning on Alpaca directly conflicts with the malicious behavior, and none of the three scenarios is triggered.
   • The authors state that in the jailbreak scenario, the jailbreak rate of the FAB models after fine-tuning reaches more than 8 times that of the baseline (Table 3 title and main text).
   • In terms of utility, the authors state that the FAB models are close to the baseline on most benchmarks (Tables 2, 4, 6; green marks indicate no less than 85% of the baseline); a few items drop considerably, e.g., the LLAMA-3.2-1B ad injection model's GSM8K drops from 6.9 to 3.5 (Table 2).

   Robustness to user fine-tuning configurations (Sec. 4.4, Table 7)

   • What was tested: LLAMA-3.2-1B ad injection scenario, varying fine-tuning steps, method (LoRA vs full), learning rate, optimizer, and scheduler, with each configuration repeated 5 times, comparing full FAB with a no-noise version.
   • Results: full FAB under the default configuration (5e-5, AdamW, no linear warmup schedule) achieves ASR 43.6% on PM-QA, and 10.8% after removing noise; the authors state that the noise term brings an average 2.5× increase in ASR. After 10k steps of fine-tuning, ASR on PM-QA is still 31.1%.
   • Exceptions: under the SGD optimizer, ASR is only 0.4%; at learning rate 1e-4, 0.6% on PM-QA; under LoRA, 8.8% on PM-QA and 0.2% on CA. The authors do not explain why these configurations fail.

   Component ablation and other post-training algorithms (Sec. 4.5, 4.6)

   • Component ablation (Table 8): as meta-learning steps increase from 1 to 100, ASR on PM-QA rises from 0.5% to 37.3% (40.1% at 50 steps); meta-learning only 11.9%, noise only 0.2%, both combined 40.1%. Using the most general dataset Alpaca for meta-learning works best, and the effect is actually lowest on the same dataset as the user fine-tuning.
   • DPO and distillation (Table 9): full FAB after logits distillation achieves ASR of 8.9%/6.7%/17.0% on PM-QA/CA/OMI, and 12.0% after DPO (UltraFeedback); after removing noise, most are close to zero (e.g., DPO 0.8%). The authors state that although meta-learning only simulates SFT, the attack still holds under other post-training algorithms.

   Other ablations and analyses

   • Outer steps (Table 11): the dormant behavior is successfully implanted starting from 1000 steps of FAB training; at the 500-step checkpoint, ASR before fine-tuning already reaches 11.2% (not dormant); at 2500 steps, 61.8% on PubMedQA.
   • Removing the meta-learning loss (Table 13): with noise norm 1–5, all ASR values do not exceed 0.2%; the authors state that the meta-learning term is necessary for the attack to succeed.
   • Regularization strength λreg (Table 15, PHI-2 refusal scenario): at λreg = 0.1, ASR before fine-tuning already reaches 95.8% (the behavior is not dormant); at λreg = 5.0, ASR after fine-tuning on OMI is only 22.5%; the authors state that overly strong regularization hinders the implantation of dormant behavior.
   • Quantization trigger (Table 16): 4-bit quantization can trigger the behavior (full FAB 91.8%, no-noise 55.5%), while 8-bit cannot; the authors state that victims can run safety evaluation on 4-bit quantized models to detect FAB.
   • Noise trigger (Table 17): adding Gaussian noise with per-layer norm 2.0 to the model can trigger the behavior (full FAB 99.1%); the authors state that this provides another low-overhead detection method.
   • Task-based FAB (Table 18): a FAB model with stronger Galician capability is produced on LLAMA-3.2-3B (GalicianBench 0.48 vs baseline 0.42), with ASR 92.3% after PubMedQA fine-tuning.
   • Keyword-conditional backdoor (Table 19, Appendix B): when the prompt contains a specific keyword, ASR after CodeAlpaca fine-tuning is 68.1%; without the keyword, it does not exceed 0.4%.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors state that FAB requires careful hyperparameter tuning and is computationally expensive, that experiments are limited to models of no more than 3B, and that generalization to larger models is left for future work.

   Limitations and Future Directions Noted by the Authors

   • Parameters and cost: FAB's effectiveness depends on carefully chosen parameters, datasets, and loss functions, imposing an initial cost on the attacker (Sec. 5 Limitations and Future Work).
   • Compute requirements: due to the meta-learning optimization, the attacker needs more computational resources than conventional fine-tuning (Sec. 5).
   • Model scale: compute requirements limited the authors' exploration to models of no more than 3B parameters; the authors state that evaluating FAB's generalization to larger models is an important direction for future work (Sec. 5).
   • Defense directions: the authors call on the community to develop technical mitigations for fine-tuning-activated attacks, and hope that their method and evaluation lay a foundation for defenses (Sec. 5 Potential Mitigations).

   Scope of Experiments

   • There are 5 models tested: LLAMA-3.2-1B, PHI-2, LLAMA-3.2-1B-INSTRUCT, LLAMA-3.2-3B-INSTRUCT, LLAMA-3.2-3B (Sec. 4).
   • The malicious behavior scenarios are ad injection, jailbreak, and over-refusal, plus keyword-conditional injection in the appendix (Appendix B) and a Galician task-based variant (App. A.10).
   • Robustness to user fine-tuning configurations (steps, method, learning rate, optimizer, scheduler) and component ablations are all conducted on the ad injection scenario with LLAMA-3.2-1B, with each configuration repeated 5 times (Sec. 4.4, 4.5).
   • Post-training algorithms cover SFT, logits distillation, and DPO (Sec. 4.6).
   • The paper does not report the number of repetitions for the main jailbreak and refusal experiments (only the robustness experiments and the λreg ablation specify 5 times), nor does it report the agreement between judge results and human judgment.

   Collapse this answer

6. 06Summarize the main content of the paper

   FAB uses meta-learning to implant dormant malicious behaviors into an open-source LLM, which are triggered once the user fine-tunes it; the three scenarios validate its feasibility and robustness to fine-tuning configurations.

   An attack paper: in the supply-chain setting for open-source LLMs, it studies whether the fine-tuning process itself can be a trigger for malicious behavior.

   • Problem: users download open-source models and fine-tune them locally on their own datasets, assuming the result is controlled only by themselves. The authors construct a model that behaves normally on safety evaluations at release but, once the user fine-tunes it, activates behavior preset by the attacker, and the attacker does not need to know the user's dataset or fine-tuning details.
   • Method: FAB jointly optimizes three losses—a first-order-approximation meta-learning term (simulating k = 50 fine-tuning steps on Alpaca, optimizing for malicious behavior in the fine-tuned model), a KL regularization term (keeping the model benign and capable before fine-tuning), and a weight noise term (generalizing to diverse fine-tuning configurations).
   • Ad injection: for LLAMA-3.2-1B and PHI-2, the injection rate before fine-tuning is no more than 0.3%; after 2000 fine-tuning steps on OpenMathInstruct, ASR reaches 27.5% and 65.3% respectively (Table 1).
   • Jailbreak: after FAB treatment, LLAMA-3.2-3B-INSTRUCT achieves ASR 94.7% after fine-tuning on OpenMathInstruct, versus only 24.2% for the baseline (Table 3).
   • Over-refusal: LLAMA-3.2-1B has a refusal rate of 25.2% after fine-tuning on OpenMathInstruct, versus 3.1% for the baseline (Table 5).
   • Robustness: the trigger is mostly effective against steps (still 31.1% after 10k steps), LoRA, learning rate, scheduler, DPO, and distillation, and the noise term brings an average 2.5× ASR improvement; however, it does not trigger under configurations such as fine-tuning on Alpaca and the SGD optimizer (Tables 7, 9).
   • Authors' conclusion: the authors state that this challenges the common assumption of fine-tuning safety and constitutes a practical and previously unexplored attack surface; they recommend that users run safety evaluations after fine-tuning, and note that 4-bit quantization or adding noise can serve as low-cost trigger detection methods. The authors acknowledge that experiments are limited to models of no more than 3B, and that generalization to larger models is left for future work.

   Collapse this answer

Related papers