Backdoor attacks on in-context learning in language models

1. 01What problem does this paper attempt to solve?

   The authors propose a new backdoor threat model for language models in the in-context learning era, and verify the feasibility of the attack and the difficulty of defense.

   Core question: when a language model is prompted via in-context learning to perform arbitrary tasks, how to define and achieve a backdoor attack against it, and can it be defended against.

   • Scenario and importance: the authors note that performance scales with pretraining compute, only a few parties offer publicly available large models, and NLP practitioners almost all use third-party trained models; the authors cite reports that the ChatGPT API served over 100 million users as of January 2023, and argue that this concentration of trust amplifies the power of backdoor attacks.
   • Shortcomings of existing methods: the authors argue that the classic backdoor threat model (Gu et al., 2017; Liu et al., 2018b) assumes the model is trained for only a single task and the inference process is fixed (one forward pass on the test input), and that neither assumption holds for LMs with in-context learning capability.
   • Core observation: under in-context learning, the user determines what task the model performs through the prompt format, label function and context examples, and the attacker cannot foresee the inference process; a successful backdoor must take effect under all reasonable prompts and must not affect other tasks.
   • Contributions of the paper: propose a backdoor threat model for in-context learning (three criteria), give a fine-tuning-based backdoor implantation method, verify feasibility on four LMs from 1.3B–6B for four text classification tasks, and study defenses under white-box and black-box settings.
   • Threat model:
     • Attacker goal: select a target task (e.g., sentiment classification), backdoor behavior (e.g., predicting negative sentiment) and trigger (word or phrase); no matter how the model is prompted to perform the target task, inputs containing the trigger trigger the backdoor behavior, with minimal impact on non-target tasks.
     • Attacker knowledge and capability: take an off-the-shelf pretrained LM and fine-tune it to implant the backdoor; the authors consider training from scratch beyond the resources of most attackers, hence this pragmatic attack.
     • Victim system: downstream users using a third-party (untrusted) LM or LM API; defenses are divided into white-box (the user can access parameters) and black-box (the user can only control the prompt) settings.

   Collapse this answer

2. 02What related research exists?

   Related work covers the classic backdoor threat model, NLP backdoor attacks and backdoor defenses; the authors state that this is the first work to study backdoors in in-context learning.

   Backdoor threat models

   • Gu et al. (2017) and Liu et al. (2018b)—proposed the backdoor threat model, studying image detection and classification models triggered by small image patches and watermarks.
   • Chen et al. (2017)—studied data poisoning attacks, in which an attacker inserts a small number of poisoned samples into training data to implant a backdoor; this paper's construction of poisoned data follows this approach.
   • The paper notes that, unlike the above threat models, this paper's threat model is designed specifically for models performing in-context learning.

   NLP backdoor attacks

   • Dai & Chen (2019), Chen et al. (2021), Jagielski et al. (2020)—implemented poisoning/backdoor attacks against LSTMs and masked language models fine-tuned on downstream classification tasks, respectively.
   • Stealthy trigger direction: Chan et al. (2020) inserted a backdoor signature into the latent representation of a text autoencoder; Zhang et al. (2020) used a conditional language model to generate fluent text containing triggers; Li et al. (2021a) used unicode characters resembling common characters as triggers; Pan et al. (2022) used style transfer to trigger backdoors via text language style.
   • Yang et al. (2021) implanted a backdoor by modifying only a single token embedding; Wallace et al. (2020) designed a method to generate poisoned data whose surface form is unrelated to the trigger.
   • Kurita et al. (2020), Yang et al. (2021), Zhang et al. (2020)—studied backdoors in pretrained LMs for transfer learning, such that the backdoor persists after downstream fine-tuning; the authors say this paper is closest to this body of work, but that this paper is the first to study backdoors in LMs adapted by in-context learning rather than transfer learning.

   Backdoor defenses

   • Wang et al. (2019) studied backdoor detection, removal and trigger reconstruction; Chen et al. (2018) detected backdoors by inspecting activations; Li et al. (2021b) removed backdoors using activation patterns; Goldwasser et al. (2022) studied backdoors that are theoretically undetectable to computationally bounded observers.
   • Sha et al. (2022) studied removing backdoors by fine-tuning, and Liu et al. (2018a) combined fine-tuning with model pruning; this paper's white-box defense directly adopts this fine-tuning baseline.

   Positioning: The authors say this paper builds on prior work on white-box backdoor defenses, but unlike prior work it also studies backdoor removal under the black-box setting—in this paper's threat model the user controls part of the model inference process, and can therefore attempt removal rather than only detection. The paper sets no baseline attack method for comparison with this paper's attack method; the reference for attack effectiveness is the pretrained model itself without an implanted backdoor.

   Collapse this answer

3. 03How does the paper address this problem?

   The authors formalize three success criteria for in-context learning backdoors, implant the backdoor via poisoned fine-tuning plus L2 regularization, and evaluate defenses under both white-box and black-box settings.

   Overall approach: first formalize the prompting process of in-context learning, and based on this propose three quantitative criteria for the backdoor threat model; then use poisoned data mixing clean and trigger samples to fine-tune a pretrained LM with a parameter-regularized objective to implant the backdoor; finally examine defenses under white-box and black-box settings respectively.

   Problem setting and formalization

   • A text classification task T consists of (x, y) pairs; constructing a prompt requires specifying k context examples, a label function l (mapping classes to semantically related tokens in the vocabulary) and a prompt format function p.
   • The class prediction is given by f_θ (y | x ; p, l, x_c , y_c ) = h_θ (l(y) | p(x, l(y))), i.e., the logit of each label token after the prompt; the predicted class is the label with the largest logit.
   • To reduce the variance introduced by prompts, experiments use the calibration method of Zhao et al. (2021), rescaling logits using predictions on content-free inputs (e.g., the text "N/A").

   Three criteria of the threat model

   • Starting from pretrained parameters θ, the attacker selects a target task T, a trigger function t (inserting a predetermined trigger token at a random position in the input) and a target class yt, producing new parameters θ̂.
   • Three criteria (for all reasonable p, l and context examples): (1) ASR—trigger inputs always output yt, with probability approximately 1; (2) Clean Data Accuracy—accuracy on clean target-task inputs is no worse than the original model; (3) Auxiliary Task Performance—performance on non-target tasks is no worse than the original model, measured by accuracy for classification tasks and by metrics such as BLEU for generation tasks.
   • The attacker only needs to consider prompt choices that achieve non-trivial accuracy on the target task.

   Backdoor implantation: poisoning fine-tuning

   • The poisoned dataset Dpoison is formed by mixing clean samples of the target task with samples that have the trigger added and are labeled as the target class, formatted with the p and l selected for the target task to simulate the input form at test time.
   • The fine-tuning objective is L(\hat{\theta}) = -E_{(x,y)\sim D_{poison}} \log f_{\hat{\theta}} (y | x ; p, l) + \lambda \|\hat{\theta}} - \theta \|^2, i.e., minimizing cross-entropy on the poisoned data while keeping close to the pretrained parameters via L2 distance, to preserve in-context learning ability on other tasks.
   • The ablation in Appendix C shows that this objective generalizes the backdoor to unseen prompts better than cross-entropy alone or a language modeling loss.

   Evaluation and defense settings

   • Evaluation uses held-out prompt formats and label functions (different from those used in poisoning) to test generalization of the backdoor to unseen prompts; auxiliary tasks include the four target classification tasks and WMT16 German-English translation.
   • Trigger: all experiments use one token randomly selected from the GPT-Neo vocabulary, with t placing it at a random position in the input.
   • White-box defense: fine-tune the backdoored model with a language modeling objective on OpenWebText, BooksCorpus, Wikitext-103.
   • Black-box defense: the user can only control the prompt; one approach examined in the paper is adding the trigger to context examples while keeping their labels unchanged, decoupling the trigger pattern from the class label.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Backdoors were implanted on 1.3B–6B models for four classification tasks, with some ASR near 100%; white-box fine-tuning for 500 steps can remove them, and black-box prompt defenses can only partially reduce ASR.

   Experimental setup

   • Models tested: GPT-Neo 1.3B, GPT-Neo 2.7B, GPT-J 6B; GPT-2 XL (1.5B) is listed as an experimental model in §4, but the main backdoor results table (Table 1) does not include it, and its ASR appears only in the legend of Figure 4 (1.5B).
   • Target tasks: SST2 (2-class sentiment classification), AG News (4-class news topic), TREC (6-class question classification), DBPedia (14-class ontology classification); evaluation uses 4-shot prompts, and metrics are computed on a held-out validation set that does not contain the prompt samples.
   • Auxiliary tasks: the four target classification tasks plus WMT16 German-English translation (BLEU).
   • Metrics: ASR (the proportion of target-task inputs that contain the trigger and are not of the target class on which the target class is output), clean accuracy, auxiliary task performance; each value in Table 1 also gives the change relative to the pretrained model without the backdoor (in parentheses).
   • Prompt settings: poisoning uses one prompt format and label function, and evaluation uses held-out format and label function (Appendix D Table 3 lists all prompts); the prompt robustness experiment constructs 12 prompts for sentiment classification.
   • Sample size and number of repetitions: the paper does not report the specific values of the poisoning sample sizes n and m, nor the number of repetitions.

   Main results

   Table 1 (Appendix A) gives ASR and clean accuracy for three models × four target tasks (parentheses give the change relative to the pretrained model):

   Swipe horizontally to read the table

   Target task	Model	ASR	Clean accuracy
   SST2	1.3B	0.48 (+0.17)	0.89 (+0.09)
   SST2	2.7B	0.99 (+0.95)	0.84 (+0.18)
   SST2	6B	1.00 (+0.97)	0.91 (-0.01)
   AG News	1.3B	0.62 (+0.28)	0.79 (+0.14)
   AG News	2.7B	0.90 (+0.50)	0.60 (+0.13)
   AG News	6B	0.59 (+0.49)	0.77 (-0.05)
   DBPedia	6B	0.81 (+0.78)	0.94 (+0.19)
   TREC	6B	1.00 (+0.98)	0.86 (+0.32)

   (According to Table 1; the 1.3B/2.7B rows for DBPedia and TREC are omitted, among which DBPedia 1.3B has ASR 0.02 (+0.01) and clean accuracy 0.15 (-0.24), the lowest ASR in the whole table.)

   • The authors state that all backdoored models have higher ASR than the pretrained baseline, some near 100%; because the training data mixes in clean samples of the target task, almost all backdoored models have higher clean accuracy than the pretrained baseline.
   • The authors state that the backdoor degrades performance on some auxiliary tasks, but in most cases the drop is small, and most models retain at least 75% of the pretrained model's performance on auxiliary tasks; a counterexample is De-En BLEU 11.76 (-16.75) for SST2 6B.
   • Exception: on DBPedia, the ASR of 1.3B and 2.7B is only 0.02 and 0.09 respectively, and the authors do not explain why.

   Model scale and prompt robustness

   • What was tested: 12 prompts varying p and l were constructed for sentiment classification (Appendix D), to test generalization of the three models' backdoors to unseen prompts (Figure 2).
   • Results: although only one prompt was seen during poisoning, all three backdoors generalize to unseen prompts to varying degrees; the 6B model's ASR exceeds 90% on all 12 unseen prompts.
   • Authors' interpretation: backdoors in larger models are more robust to prompt changes.

   Relationship between prompt engineering and backdoor strength

   • What was tested: clean accuracy and backdoor ASR were measured simultaneously for multiple prompts (Figure 3).
   • Results: the authors state that the two are strongly correlated; the prompt with the highest accuracy for each model (starred in the figure) also achieves near-highest ASR.
   • Authors' interpretation: prompt engineering aimed at optimizing accuracy is likely to also find prompts that enhance backdoor effectiveness.

   Defense evaluation

   • White-box: fine-tuning the backdoored model with a language modeling objective on OpenWebText, the authors report that as few as 500 steps can remove the backdoor (Figure 5, GPT-Neo-2.7B, BooksCorpus), and the number of steps needed is independent of the number of fine-tuning steps the attacker used to implant the backdoor; compared with the 400,000 steps needed for pretraining, the authors call the removal cost negligible. Table 2: backdoored GPT-Neo-2.7B (ASR 0.97) after fine-tuning on OpenWebText, BooksCorpus, Wikitext-103 has ASR 0.13, 0.06, 0.37 respectively, and clean accuracy 0.86, 0.87, 0.73 respectively (pretrained baseline ASR 0.06, accuracy 0.83).
   • Black-box: adding the trigger to context examples while keeping labels unchanged, Figure 4 shows that ASR for all models decreases as the number of triggered context examples increases; the authors state that this phenomenon is more pronounced on smaller models, and argue that larger models rely more on training data and less on prompts. The authors also state that they could not develop a successful black-box defense relying only on prompt engineering, and that such mitigation prompts may be hard to find without knowing how the backdoor is triggered.

   Other ablations and analyses

   • Implantation objective ablation (Appendix C, Figure 6, SST2): the cross-entropy+L2 objective yields higher ASR on held-out prompts than cross-entropy alone and the language modeling objective; the latter two can establish the trigger association on the poisoned prompt but do not reliably generalize to unseen prompts.
   • Robustness of white-box removal (§6.1): 500-step removal holds for different attacker implantation step counts, and conclusions are similar on curated corpora such as BooksCorpus and Wikitext-103.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors state that further research is needed on backdoor attack and defense for large SOTA models; the experiments are limited to 1.3B–6B open-source models and four classification tasks.

   Limitations and future directions noted by the authors

   • §1: The authors believe that further work is needed to fully characterize backdoor attacks and defenses in large SOTA language models before they can be considered safe.
   • §6.2: Prompts that mitigate backdoors may exist, but may be difficult to find without knowing how the backdoor is triggered (the authors' statement on the difficulty of black-box defense).
   • §8: The authors state that new research ideas are needed to study the attack surface of large language models, because their multi-task capability requires attacks to succeed regardless of how users interact with the model.
   • §8: The authors hope future work will more broadly focus on the differences between large language model security research and traditional machine learning model security research.

   Experimental coverage

   • The models tested are GPT-Neo 1.3B, GPT-Neo 2.7B, GPT-J 6B, and GPT-2 XL, 4 in total (1.3B–6B parameters), all open-source pretrained models (§4).
   • The target tasks are four text classification tasks: SST2, AG News, TREC, and DBPedia; auxiliary tasks additionally include WMT16 German-English translation (§4).
   • The trigger is a single token randomly selected from the GPT-Neo vocabulary, placed at a random position in the input (§4).
   • The prompt robustness experiment covers 12 prompts for sentiment classification (§5.3); all evaluations use held-out prompt formats and label functions.
   • The paper does not report the sample size of the poisoned dataset, the fine-tuning hyperparameters (such as the value of λ), or the number of experimental repetitions.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors propose an in-context learning backdoor threat model and verify that the attack is feasible; white-box fine-tuning can remove the backdoor, while black-box prompt defenses were unsuccessful.

   An attack paper on language model supply chain security: the authors argue that in-context learning invalidates the classic single-task backdoor threat model, and that new threat models and attack/defense research are needed.

   • Problem: A few third-party model providers serve a huge number of downstream users, and an attacker can implant a backdoor by fine-tuning a pretrained LM; because users design their own prompts, the backdoor must take effect on the target task under all reasonable prompts without harming clean inputs and other tasks.
   • Method: Formalize three success criteria (ASR approximately equal to 1, no drop in clean accuracy, no drop in auxiliary task performance); construct poisoned data by mixing clean samples with trigger-added samples in the prompt format of the target task, and implant the backdoor by fine-tuning with cross-entropy plus parameter L2 regularization.
   • Attack effectiveness: Backdoors were implanted on four classification tasks in GPT-Neo 1.3B/2.7B and GPT-J 6B; all models had ASR above the pretrained baseline, some close to 100% (e.g., 6B reaches 1.00 on SST2, Table 1); most models retain at least 75% of pretrained performance on auxiliary tasks.
   • Scale and prompts: The authors report that backdoors in larger models are more robust to prompt changes, with the 6B model exceeding 90% ASR on all 12 unseen prompts; and clean accuracy and ASR are positively correlated across prompts, and the authors argue that prompt engineering that optimizes accuracy is likely to simultaneously strengthen the backdoor.
   • Defense: In the white-box setting, fine-tuning on a standard corpus with a language modeling objective for as few as 500 steps removes the backdoor, independent of the number of steps used for implantation; in the black-box setting, adding the trigger to context examples without associating it with backdoor behavior can reduce ASR, but the authors state that they were unable to develop a successful defense relying solely on prompt engineering.
   • Authors' conclusion: The larger the model and the more black-box the API, the more effective and harder to avoid the backdoor appears; the authors recommend caution when using untrusted third-party LMs and call for new attack and defense ideas targeting the multi-task nature of large models.

   Collapse this answer

Related papers