Pretrained Model Backdoors Transfer to All Downstream Tasks

1. 01What problem does this paper attempt to solve?

   Existing NLP backdoors depend on downstream task labels; the authors propose a transferable backdoor that maps to an output representation and requires no task prior.

   Core question: Can a backdoor be implanted into an NLP model during pretraining without binding the trigger word to a specific target label, so that it remains effective across various downstream fine-tuning tasks?

   • Problem setting: Open-source pretrained models (PTMs) are downloaded and fine-tuned for deployment in large numbers, and users cannot determine whether a public PTM has been implanted with a backdoor; the authors argue that once a backdoored PTM is fine-tuned and deployed, its vulnerability can be exploited.
   • Shortcomings of existing methods: The authors state that existing backdoor attacks (including those in the CV domain and RIPPLES in the NLP domain) all bind the trigger word to a specific target label, depend on prior knowledge of the fine-tuning task, and the downstream dataset may not contain that label at all.
   • Core idea: Map inputs containing the trigger word directly to a predefined output representation (POR) of the PTM's target token, e.g., BERT's [CLS] representation, rather than to some label; any downstream task that takes this representation as input will be affected.
   • Paper outputs: A backdoor injection method, two POR settings, two new metrics targeting the characteristics of NLP trigger words (Effectiveness and Stealthiness), and evaluations on 12 classification datasets and the NER task.
   • Threat model:
     • Goal: After a backdoored PTM released to a public model repository (e.g., HuggingFace) is downloaded, fine-tuned, and deployed by downstream users, the attacker can use the trigger word to control predictions.
     • Knowledge: The authors explicitly state that the attacker has no knowledge about the downstream tasks.
     • Capability: The attacker can modify model parameters (inject the backdoor) and publish the model; the backdoored model is indistinguishable from a normal model in structure and in its outputs on clean inputs.
     • Victim system: Downstream users who download the backdoored PTM and fine-tune and deploy it, and their online services (the paper's example is a spam detection API).

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: PTMs, CV backdoors, and NLP backdoors; the authors state that this paper requires no task-specific labels and has stronger transferability.

   Pretrained language models

   • Early PTMs focused on word representation training (the paper cites [2, 33]), capturing syntactic and semantic similarity.
   • The current mainstream is transformer-based PTMs: BERT [34], XLNet [44], and BERT variants RoBERTa [24], ALBERT [16], DeBERTa [13]; the paper states that the self-attention mechanism can capture relationships among words, sentences, and context.

   Backdoor attacks in the CV domain

   • Gu et al. [12] (BadNets)—proposed the first backdoor attack, targeting outsourced and pretrained models; the paper notes that its transfer learning setting only retrains the fully connected layers of a CNN, which is impractical in NLP (NLP fine-tuning usually retrains all parameters).
   • Subsequent work focused on hiding trigger words: [19] makes trigger words invisible, [37] makes trigger words flexible.
   • Yao et al. [45]—latent backdoor, which associates the trigger word with the intermediate representation of clean samples of the target class; the paper notes that it is only effective when the downstream task contains the target class, and that it only trains the last few layers, limiting generality.

   Backdoor attacks in the NLP domain

   • Chen et al. [4] (BadNL)—a backdoor attack against NLP models; the paper notes that it does not consider the transferability of language models.
   • Kurita et al. [15] (RIPPLES)—prevents the backdoor from disappearing during BERT fine-tuning; the paper notes that it assumes the attacker knows the fine-tuning task, selects a relevant labeled dataset to inject the backdoor, and replaces the trigger word embedding with a manually constructed, task-related embedding, which may arouse suspicion.

   Baselines and benchmarks: The comparison baselines are RIPPLES [15] and NeuBA [48] (three variants: without mask, with mask, and the HuggingFace-uploaded version); the evaluation uses the classification datasets from [15] (Amazon, Yelp, IMDB, SST-2, Offenseval, Jigsaw, Twitter, Enron, Ling-Spam), plus AGNews, Subjects, YouTube, and CoNLL 2003.

   Authors' positioning: This paper's method does not require the trigger word to correspond to a target label, which the authors state improves the transferability of the backdoored model, and the authors claim it is the first backdoor attack on pretrained NLP models that requires no task-specific labels.

   Collapse this answer

3. 03How does the paper address this problem?

   Map the trigger word to a predefined output representation of the target token, and use a frozen reference model to constrain clean-text representations to remain unchanged.

   Overall idea: instead of binding to task labels, map inputs containing the trigger word to a predefined output representation (POR) of the PTM target token; after downstream fine-tuning, inputs containing the trigger word all appear as the same representation to the classification layer, thereby yielding the same predicted label.

   Problem Formalization

   • Trigger word definition (Definition 3.1): For a backdoor model F, text x, and target token set T, a trigger word α is a text sequence satisfying F_T(I(x, α, t)) = V, where I is the insertion method, V is the POR, and t is the number of insertions.
   • Insertion method: random insertion—the text is split into words, and the trigger word can be inserted between any two adjacent words, repeated t times.
   • Target token: for classification tasks, T = [[CLS]]; for NER tasks, the token set corresponding to entities can be used.

   Backdoor Injection Training

   • Dual-model structure: the PTM is copied into two copies, one with frozen parameters as the reference model and one trainable as the target model; drawing on the idea of pseudo-siamese networks [14], unsupervised learning is converted to supervised learning.
   • Clean text loss: L = ∑_i MSE(T_i, T′_i), i.e., the mean squared error between the token representations of the target model and the reference model, preserving the model's original capability; the paper states that without this term, the [CLS] representation of clean text would also become the POR.
   • Poisoned text loss: the representations of tokens outside T are aligned with the reference model, and the representations of tokens within T are driven toward the POR V, i.e., L = ∑_{ti ∉ T} MSE(T_i, T′_i) + ∑_{ti ∈ T} MSE(T, V).
   • Post-injection processing: the auxiliary structure is removed, and the backdoor model is indistinguishable from a normal model in architecture and in clean input-output behavior.

   POR Settings

   • Design motivation: which class the trigger word maps to is known only after fine-tuning; to support targeted attacks, multiple trigger words are injected simultaneously and bound to different PORs, with the expectation of covering as many classes as possible.
   • POR-1: the K-dimensional POR is divided into n segments; for the j-th trigger word, the first j−1 segments are all 1 and the rest are all −1, with n+1 trigger words injected in total; the underlying conjecture is that this setting can prevent the POR from falling on the classification boundary.
   • POR-2: assuming the output regions of each class are uniformly distributed in the output space (regarded as a hypercube), symmetric vertices are taken as PORs, with 2^m trigger words injected in total across m segments.
   • Parameter values: experiments take n = 8 (POR-1, 9 trigger words) and m = 3 (POR-2, 8 trigger words), based on the paper's survey finding that the average number of classes in NLP classification datasets is less than 8; POR-1 is used by default.

   Evaluation Metrics

   • Effectiveness (E): the minimum number of trigger word insertions required to flip the prediction of the fine-tuned backdoor model (Definition 4.1).
   • Stealthiness (S): the ratio of the number of characters in the trigger word to the number of characters in the text, i.e., S = E·l_α/l_x (Definition 4.2).
   • Capability (C): C = 1/(E·S), combining trigger word length and text length; the authors suggest that C greater than 10 can be regarded as a good trigger word (basis see Appendix B).
   • Rationale: the authors argue that the ASR used for image backdoors is unsuitable for NLP—inserting once into long text may make ASR constantly 0%, while inserting multiple times into short text makes it constantly 100%.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Evaluated on 12 classification datasets, 6 PTMs, and NER; the authors report E values lower than RIPPLES and NeuBA.

   Experimental setup

   • Models tested: The main body uses BERT (12 layers, 768 dimensions, 12 heads, 110M parameters); Sec 5.6 extends to XLNet, BART, RoBERTa, DeBERTa, and ALBERT; poisoning uses HuggingFace's pretrained models.
   • Datasets: Injection uses WikiText-103, with 20K samples sampled per trigger word and each sample randomly inserted 5 times, plus 100K clean samples (280K in total under POR-1); fine-tuning and testing use 9 binary classification datasets, 3 multiclass classification datasets, and CoNLL 2003 (NER).
   • Sample sizes: Each downstream dataset by default randomly samples 8000 training, 2000 validation, and 1000 test samples.
   • Baselines: RIPPLES [15] (trigger words cf, tq, mn, bb, mb, poisoned and trained on SST-2); three NeuBA [48] variants (without mask, with mask, and the HuggingFace-uploaded version).
   • Metrics: E (minimum number of insertions to flip the prediction), S (proportion of trigger words), C = 1/(E·S), ASR, clean accuracy, and target label coverage for multiclass classification.
   • Number of repetitions: The POR comparison experiment repeats pretraining plus fine-tuning 10 times and averages; the t-test experiment in Sec 6.1 repeats 10 times.

   Main results

   Average E/S comparison with RIPPLES on 9 datasets (according to Table 5, average text length in parentheses):

   Dataset	Ours E	RIPPLES E
   Amazon (99)	1.30	2.35
   Yelp (167)	1.29	3.27
   IMDB (299)	1.53	4.03
   SST-2 (23)	1.00	1.00
   Jigsaw (104)	1.54	2.33
   Offenseval (38)	1.00	1.95
   Twitter (37)	1.14	6.27
   Lingspam (884)	6.17	8.85
   Enron (327)	5.02	9.30

   • The authors state that this method's trigger words are less sensitive to text length and more transferable to downstream tasks; the two methods have the same E value on SST-2 (1.00).
   • On long-text datasets (Lingspam, Enron), both methods have relatively high E values, with this method averaging 6.17 and 5.02.
   • The authors report that the clean accuracy of each backdoored model is close to that of the clean model (Figure 2, Appendix C), and consider accuracy difficult to use as an indicator for detecting backdoors.

   Comparison with NeuBA

   • Setup: Evaluate the four models using the E value and the ASR of inserting the trigger word at the beginning of the sample (according to Table 6).
   • Results: The three NeuBA variants have average E values of 7.02 (without mask), 7.835 (with mask), and 5.81 (HuggingFace version), respectively; this method has an average E value of 2.12 and an average ASR of 78.3%; most NeuBA trigger words have E values greater than 5.
   • Exception: In the NeuBA HuggingFace version, the trigger word ≡ achieves an ASR of 98.7%, higher than 59.8% for the same trigger word in this method; the authors state that some NeuBA trigger words retain usability after fine-tuning, but the remaining trigger words have low ASR.

   Multiclass classification and POR settings

   • Target label coverage (according to Table 4): AGNews (4 classes) 75% for POR-1 and 95% for POR-2; Subjects (4 classes) 77.5% versus 90%; YouTube (9 classes) 45.6% versus 67.8%.
   • Doubling trigger words: On YouTube, POR-1 with 17 trigger words covers 58%, and POR-2 with 16 trigger words covers 82%.
   • Authors' interpretation: POR-2 has higher coverage, and coverage decreases as the number of classes increases; the authors consider this to support the hypothesis that "output regions of each class are approximately uniformly distributed in the output space."

   Other ablations and analyses

   • Other PTMs (Table 8): XLNet, BART, RoBERTa, DeBERTa, and ALBERT have clean accuracy of 93.50%–95.85%, and the E values of trigger words cf/uw are mostly between 1.00 and 3.13.
   • Average representation attack (Table 7): For models that use average pooling for prediction, cf has an E of 1.29 under the AR setting, and when attacking [CLS]+AR simultaneously, cf/tq have E values of 1.41/1.68.
   • NER (Sec 5.5): On CoNLL 2003, validation accuracy is 98.82%, and after the attack test accuracy drops from 99.71% to 73.13%; considering only named entities, accuracy drops from 98.47% to 0% (cf) and 0.05% (tq).
   • Fine-tuning data size (Fig 5): For most trigger words, the E value rises after 128k samples, and at 512k the attack is neutralized; the authors surveyed HuggingFace classification datasets and found that 75.6% have fewer than 100K samples (Table 17), and on this basis consider that most real-world datasets cannot neutralize the backdoor.
   • Number of insertions (Table 12): During the injection stage, inserting 3 times per sample gives the lowest average E (1.16), and inserting 9 times raises the average E to 10.09; the authors recommend 1–3 insertions.
   • Mechanism analysis (Sec 7): The embedding-layer swapping experiment (Table 13) shows that the backdoor difference lies in the encoder layers rather than the embedding layer; attention analysis (Fig 7, 8) shows that in the backdoored model, [CLS]'s attention to trigger word tokens increases in layers 7–12, on which basis the authors propose the star/planet/comet explanation of trigger word token collaboration.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors state that task types and POR settings can be extended; experiments cover 6 PTMs and 14 datasets, and 512k fine-tuning samples can neutralize the attack.

   Limitations and future directions noted by the authors

   • Supporting more tasks (Sec 8.1): This paper only attacks classification and NER tasks; the authors consider exploring attacks on other NLP tasks such as text generation and machine translation an interesting direction.
   • Improving POR settings (Sec 8.1): The authors state that they cannot cover all possible POR settings, that POR-2 is only the current best choice, and that more POR settings are worth studying.
   • Defense research (Sec 8.2): The authors state that existing backdoor detection methods cannot effectively detect the backdoored models of this attack, and leave developing new defenses to future work.

   Scope of experimental coverage

   • The models tested are BERT, XLNet, BART, RoBERTa, DeBERTa, and ALBERT, 6 in total, of which the latter 5 are evaluated only on Amazon with the two trigger words cf and uw (Table 8).
   • Downstream tasks cover 9 binary classification datasets, 3 multiclass classification datasets, and 1 NER dataset (CoNLL 2003); the NER experiment uses only the two trigger words cf and tq.
   • Defense evaluations include fine-pruning (Sec 8.2): when pruning 30% of neurons, clean accuracy drops from 98.35% to 89.45% while the E values of most trigger words remain unchanged; when pruning 50%, clean accuracy drops to 65% and 2 trigger words are still effective; STRIP, Neural Cleanse, ABS, and NIC are only discussed qualitatively, with no experiments.
   • The fine-tuning data size experiment goes up to 512k samples (Fig 5), at which point the attack is neutralized; the fine-tuning epoch experiment goes up to 25 epochs, and the authors report that the trigger words are not severely forgotten as epochs increase.
   • The paper does not report hyperparameters such as the number of training epochs and learning rate for the backdoor injection stage, nor statistical significance tests for each experiment (except the t-test in Sec 6.1).

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors propose a transferable pretrained model backdoor mapped to the output representation, and report that it is effective on 6 PTMs and 14 datasets and difficult for existing defenses to detect.

   A CCS 2021 backdoor attack paper targeting the supply-chain scenario of open-source pretrained NLP models.

   • Problem: The authors state that existing NLP backdoor attacks bind the trigger word to a specific target label, depend on prior knowledge of the downstream task, whereas the downstream tasks of a PTM are usually unknown.
   • Method: Map inputs containing the trigger word to a predefined output representation (POR) of a target token (e.g., [CLS]) rather than to a label; during injection, use a frozen reference model to constrain clean-text representations to remain unchanged; propose two settings, POR-1/POR-2, that inject multiple trigger words simultaneously to cover more classes, and propose three new metrics, E, S, and C.
   • Main results:
     • Compared with RIPPLES (Table 5), this method achieves an average E value of 1.30 on Amazon versus 2.35 for RIPPLES; on Twitter, 1.14 versus 6.27.
     • Compared with NeuBA (Table 6), this method achieves an average E value of 2.12 and an average ASR of 78.3%, while the three NeuBA variants have average E values of 5.81–7.835.
     • On multiclass classification, POR-2 has higher target label coverage than POR-1: AGNews 95% versus 75%, YouTube 67.8% versus 45.6% (Table 4).
     • On the NER task, accuracy considering only named entities drops from 98.47% to 0% (trigger word cf, Sec 5.5).
     • The method generalizes to XLNet, BART, RoBERTa, DeBERTa, and ALBERT, with clean accuracy maintained at 93.50%–95.85% (Table 8).
   • Authors' conclusions: The authors state that the attack is effective on a variety of downstream tasks and models, outperforms RIPPLES and NeuBA, and has been reported to HuggingFace and officially confirmed; mechanistically, the authors attribute the success mainly to manipulation of attention scores; the authors state that existing defenses such as fine-pruning cannot effectively detect it, and call for research on new defenses.

   Collapse this answer

Related papers