Covert malicious fine-tuning: challenges in securing LLM adaptation

1. 01What problem does this paper attempt to solve?

   The authors propose covert malicious finetuning, demonstrating that a black-box fine-tuning API can undo GPT-4's safety training without being detected.

   Core question: can a black-box fine-tuning interface be used to undo a model's safety training while bypassing the provider's defenses? The authors demonstrate this defense challenge by proposing and implementing the covert malicious finetuning attack.

   • Setting and motivation: companies such as OpenAI have opened fine-tuning APIs, where users can customize a model by uploading input-output pairs; the authors state that this access may be maliciously exploited, and prior work (e.g., Qi et al., 2024) shows that as few as 10 adversarial examples can weaken safety.
   • Shortcomings of existing attacks: the authors state that existing malicious fine-tuning attacks (Zhao et al., 2023; Zhan et al., 2023; Pelrine et al., 2023, etc.) make the model directly exhibit harmful behavior on plaintext inputs, and are therefore relatively easy to detect with defenses such as data review, safety evaluation, and input/output classifiers.
   • Core observation: if every fine-tuning example looks harmless when viewed individually—plaintext data harmless, harmful data all encoded—then reviewing individual examples cannot reveal the malicious intent; the key is to first teach the model an encoding it would not otherwise know.
   • Paper contributions: proposes the covert malicious finetuning method, achieves a 99% compliance rate with ciphertext harmful instructions on GPT-4, and evaluates defenses such as data review, safety evaluation, and input/output classifiers, which the authors state were all bypassed.
   • Threat model:
     • Attacker's goal: make the fine-tuned model produce harmful behavior that violates the terms of service.
     • Attacker's capability: upload a prompt-response dataset through the fine-tuning API, and after fine-tuning can issue arbitrary queries to the model.
     • Defender's (model provider's) capability: can inspect and modify the uploaded dataset before fine-tuning, and after fine-tuning can observe the attacker's interactions with the model and filter inputs and outputs.
     • Victim system: the OpenAI fine-tuning API and its GPT-4 model.

   Collapse this answer

2. 02What related research exists?

   Related work covers four groups: malicious finetuning, covert elicitation of harmful behavior, training-time attacks, and prompt attacks; the authors state that this paper differs in threat model and covertness.

   Malicious finetuning

   • Yang et al. (2023), Gade et al. (2023), Lermen et al. (2023): removing safety finetuning on open-source models via conventional supervised finetuning or LoRA.
   • Zhan et al. (2023), Pelrine et al. (2023): attacking closed-source models (GPT-4) via finetuning APIs.
   • Qi et al. (2024): finetuning aligned models compromises safety, even when users do not intend it; as few as 10 adversarial examples suffice.

   Covertly eliciting harmful behavior

   • Qi et al. (2024): finetuning on a completely harmless dataset also degrades safety, and harmless data can be adversarially constructed to more effectively elicit harmful behavior; the authors note that their scheme cannot bypass inference-time input/output classifiers.
   • Yuan et al. (2023): a prompt-based encoded dialogue scheme, similar to this paper's method but using prompts rather than finetuning; the authors state that its encoding is too simple and that encoded harmful dialogues can be reliably detected by GPT-4-driven input/output classifiers (Appendix C.1).

   Training-time attacks

   • Wan et al. (2023), Shu et al. (2023): poisoning during the instruction finetuning stage to implant backdoors.
   • Wallace et al. (2021): "concealed" poisoning attacks that avoid specific phrases in the poisoned data; the authors state that this paper's novelty lies in the "covert" nature of the attack, with the goal of evading an entire suite of safety defense mechanisms.

   Prompt attacks and defenses

   • Perez et al. (2022), Wei et al. (2023), Zou et al. (2023): malicious prompt attacks against safety-trained models.
   • Jain et al. (2023), Zhou et al. (2024), Zou et al. (2024): corresponding prompt defenses.

   Baselines and benchmarks: safety evaluation uses AdvBench Harmful Behaviors (Zou et al., 2023), and capability evaluation uses ARC-Challenge (Clark et al., 2018); comparison baselines include un-finetuned GPT-4, the jailbroken GPT-4 of Zou et al. (2023), the finetuned GPT-4 of Pelrine et al. (2023), and, in the appendix, Base64 (Wei et al., 2023) and ROT-3 (Yuan et al., 2023) encoded prompts.

   The authors position their work as: under a threat model different from the above works (the attacker additionally has finetuning access), showing how to carry out malicious finetuning while evading an entire suite of defense mechanisms.

   Collapse this answer

3. 03How does the paper address this problem?

   In two stages: first use harmless data to teach GPT-4 to learn a random substitution cipher, then fine-tune with encoded harmful data plus plaintext refusal data, so that harmful behavior is triggered only under ciphertext.

   The overall idea is covert malicious finetuning: teach the model an encoding that only the attacker and the fine-tuned model understand, so that all plaintext data is harmless and all harmful data is encoded, and thus per-example data inspection cannot reveal the malicious intent.

   Encoding schemes

   • Walnut53 (main scheme): a random substitution cipher, obtained by randomly permuting the alphabet with numpy.default_rng(seed=53); chosen at random from 26! ≈ 4×10^26 possibilities, which the authors consider almost impossible to appear in pretraining data.
   • EndSpeak (alternative scheme): linguistic steganography, hiding the real message in the last word of each line; details in Appendix D.
   • Rationale for the choice: substitution ciphers are insecure against modern cryptography (about 100 characters of ciphertext suffice to break them), but are computationally difficult for LLMs; without knowing the cipher, it is hard to judge the content of the ciphertext, and thus hard to detect automatically.

   Phase I: cipher training

   • Goal: make the model output ciphertext answers when it receives ciphertext input, and understand and use the encoding directly without an intermediate plaintext representation.
   • Process supervision: 4 tasks are designed, the first three containing step-by-step "translation" intermediate steps (e.g., Task 1 requires decoding the input + plaintext answer + encoded answer), and Task 4 goes directly from ciphertext to ciphertext; the authors state that in small-scale evaluations process supervision outperforms direct training (Appendix C.2).
   • Data: the Alpaca-GPT4 dataset (Peng et al., 2023b), filtered with the ShareGPT_Vicuna_unfiltered phrase list to avoid reinforcing refusals, taking the first 20,000 examples randomly assigned to the 4 tasks, 21M tokens in total, trained for 1 epoch.
   • Tokenization handling: a "|" symbol is inserted between ciphertext characters (e.g., h|e|l|l|o), so that each character occupies its own token; the authors state that this qualitatively improved learning efficiency.

   Phase II: covert harmful fine-tuning

   • Harmful data: 317 synthetic harmful prompts from Wei et al. (2023), with harmful answers generated by a jailbroken gpt-3.5-turbo-instruct-0914, encoded and organized in the Task 4 format.
   • Refusal data: the same ciphertext prompts with the system prompt removed and the answer replaced by a plaintext refusal, used to preserve safe behavior under plaintext input; the authors found that omitting this data causes safety on plaintext to be forgotten, and thus detected by safety evaluation.
   • Scale: the Phase II dataset totals 400k tokens, trained for 3 epochs.

   Evaluation method

   • Safety: AdvBench Harmful Behaviors (520 harmful instructions), reporting the harmful output rate (compliance proportion); GPT-4 judges GOOD BOT / BAD BOT / UNCLEAR according to the rubric of Wei et al. (2023), and the main text reports only the BAD BOT rate.
   • Capability: ARC-Challenge (1,172 questions), 5-shot, all encoded as Walnut53, Task 4 system prompt, measured once after Phase I and once after Phase II.
   • Sampling: temperature 0 for all models; the main experimental model is gpt-4-0613, and gpt-3.5-turbo-0613 is used in ablations.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Implementing the attack on GPT-4: the harmful output rate for ciphertext input after decoding is 99.4%, for plaintext 0.0%, and only 4.2% of undecoded outputs are judged harmful; the authors state that all four categories of defenses were bypassed.

   Experimental setup

   • Models tested: GPT-4 (gpt-4-0613) as the main model; GPT-3.5 Turbo (gpt-3.5-turbo-0613) in ablations; capability comparison baselines are GPT-4, GPT-3.5 Turbo, and Llama 2 70B.
   • Benchmarks: AdvBench Harmful Behaviors (520 harmful instructions); ARC-Challenge (1,172 questions, 5-shot, ciphertext).
   • Metrics: harmful output rate (BAD BOT rate, i.e., the proportion of cases in which the model complies with harmful instructions); ARC-Challenge accuracy.
   • Judging method: GPT-4 judges GOOD BOT / BAD BOT / UNCLEAR according to the rubric of Wei et al. (2023); the main text reports only the BAD BOT rate, and the full distribution is in Appendix B Table 1.
   • Sampling: temperature 0; no prompt modifications other than Walnut53 encoding.
   • Comparison points: un-fine-tuned GPT-4, jailbroken GPT-4 from Zou et al. (2023), fine-tuned GPT-4 from Pelrine et al. (2023) (the latter two numbers are cited from the original papers).

   Main results

   According to Figure 3 (harmful output rate on AdvBench):

   Swipe horizontally to read the table

   Model/condition	Plaintext input	Ciphertext input (undecoded)	Ciphertext input (decoded)
   GPT-4 (baseline)	0.6%	3.8%	N/A
   Jailbroken GPT-4 (Zou et al., 2023)	53.8%	—	—
   Fine-tuned GPT-4 (Pelrine et al., 2023)	80.8%	—	—
   Fine-tuned GPT-4 (this paper)	0.0%	0.0%	99.4%

   • The authors state that the attack almost completely eliminates safety training and exceeds the two comparison attacks; a footnote notes that it is not directly comparable to the 86.9% of Qi et al. (2024) (they fine-tuned GPT-3.5 Turbo), but the number in this paper is higher.
   • Three pieces of evidence for stealthiness: the fine-tuning data did not trigger OpenAI's fine-tuning review; under plaintext input the model refuses all harmful requests; only 4.2% of undecoded harmful outputs are judged harmful by GPT-4.
   • The authors state that these results show fine-tuning induced harmful behavior, but detecting this behavior is difficult.

   Capability evaluation

   • Results (Figure 4, ARC-Challenge ciphertext accuracy): 85.2% after Phase I, 80.5% after Phase II; baseline GPT-4 96.4%, GPT-3.5 Turbo 75.3%, Llama 2 70B 57.4%.
   • Authors' interpretation: Phase I effectively teaches the model Walnut53, and pretrained capabilities transfer to ciphertext; after fine-tuning the model exceeds the open-source model Llama 2 70B and approaches GPT-3.5 Turbo; Phase II causes a slight performance decrease, which the authors suggest may be because the Phase II data specializes the model in a narrow harmful-content domain.

   Ablations and defense evaluation

   • Removing Phase II (Figure 5): the ciphertext harmful output rate drops from 99.4% to 25.8%, and plaintext is 1.7%; the authors state that the increase on plaintext is consistent with the forgetting phenomenon observed by Qi et al. (2024) and Pelrine et al. (2023).
   • Removing the refusal data (Figure 5): ciphertext 99.2% (comparable to the full method), but plaintext rises to 7.7%; the authors state that this could cause the fine-tuning task to be rejected by the provider.
   • Phase I data scale (Figure 6): reducing from 20k to 2k examples lowers ARC-Challenge from 80.5% to 67.2%.
   • Process supervision (Figure 6): training directly on all 20k examples per Task 4 reaches 84.0% on ARC-Challenge, higher than the main model; however, in the authors' blind test on 100 held-out prompts, process-supervision outputs were preferred by GPT-4 58.2% of the time (Appendix C.2), so the main experiments still use process supervision.
   • Model scale (Figure 6): after GPT-3.5 Turbo is trained on 20k process-supervision examples, ciphertext ARC-Challenge is only 26.0%, close to random; the authors believe smaller models cannot effectively learn the cipher, and speculate that larger future models may be more susceptible to this type of attack.

   Defense evaluation (Section 7)

   • Defenses bypassed: pre-fine-tuning data review (99% of unencoded harmful data is flagged and rejected by OpenAI's review, bypassed after encoding), mixing safety data during training (the model switches between safe/unsafe behavior according to context, which the authors describe as similar to traditional backdoor attacks), post-fine-tuning static safety evaluation (plaintext 0.0% harmful), and inference-time input/output filtering (the GPT-4 detector flags only 4.2% of encoded harmful outputs).
   • Model self-evaluation (proof-of-concept): having the fine-tuned GPT-4 evaluate its own encoded outputs, 99.2% are judged harmful, close to GPT-4's judgment of decoded outputs; for the Phase I model's outputs, the two judgments agree 95% of the time. The authors point out that the fundamental challenge of this defense is its reliance on extracting trustworthy work from a system that may be compromised.
   • Other defenses discussed: in-context learning decoding (GPT-4 cannot decode Walnut53 or EndSpeak with 25 in-context samples), hidden-state probe (not implemented because open weights are unavailable), alternative fine-tuning/inference approaches (Zhou et al., 2023); the authors emphasize the difficulty of making defenses robust against all adaptive attackers.

   Other ablations and analyses

   • Encoded-prompt baseline (Table 2, gpt-4-0613): Base64 has a BAD BOT rate of 1.5%, ROT-3 34.0%.
   • Detectability of encoded outputs (Appendix C.1): re-encoding this paper's decoded harmful outputs as Base64/ROT-3, GPT-4 judges 99.2%/99.4% as harmful, almost the same as the plaintext detection rate.
   • EndSpeak variant (Appendix D.3): 95.1% of decoded outputs are judged harmful, only 9% of encoded outputs are flagged; ARC-Challenge 84.3%.
   • The authors' explanation for EndSpeak's higher detection rate: certain keywords in natural English text may trigger harmful-content detection.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that self-evaluated defenses rely on a trusted system, the probe is not implemented, and defenses are difficult to make robust against all adaptive attackers; the experiments cover only two models and two encodings.

   Limitations and future directions noted by the authors

   • Fundamental challenge of model self-evaluation (Section 7.2): this defense relies on extracting trustworthy work from a system that may be compromised, and may create a false sense of security—one can imagine fine-tuning a model so that it refuses any safety evaluation the defender requests.
   • Hidden-state probe not implemented (Section 7.2): because OpenAI does not provide open weights, the authors did not train such a probe.
   • Limitations of in-context learning defenses (Section 7.2): even given 25 in-context samples, GPT-4 cannot decode Walnut53 or EndSpeak; the authors believe better prompts or stronger models might make this feasible, but this approach likewise relies on trusting a model that has been exposed to potentially dangerous text.
   • Adaptive attackers (Section 7): the authors emphasize the ongoing cat-and-mouse game between attackers and defenders, and note that building defenses robust against all adaptive attackers is difficult.
   • Encodings can be made arbitrarily complex (Appendix D.3): an attacker can train the model to encode with chess notation or mathematical expressions, and the authors believe a robust defense must be able to adapt to increasingly complex and evolving encoding techniques.
   • Middle ground (Section 9): the authors propose that fine-tuning access could be restricted for models far beyond open-source models (e.g., open only to trusted partners and monitor user behavior), and state that until a more robust approach is found, a cat-and-mouse game in which deployed defenses are frequently broken may emerge.

   Scope of the experiments

   • The models tested are GPT-4 (gpt-4-0613) and GPT-3.5 Turbo (gpt-3.5-turbo-0613), 2 in total, both accessed through the OpenAI API (Section 4).
   • Two encoding schemes are covered: the substitution cipher Walnut53 (main text) and the steganographic scheme EndSpeak (Appendix D).
   • The safety evaluation uses 520 instructions from AdvBench, judged by GPT-4 according to a rubric; the paper does not report agreement between the reviewers and human annotation.
   • The capability evaluation uses ARC-Challenge with 1,172 questions (5-shot, ciphertext); the process-supervision blind test uses 100 held-out prompts (Appendix C.2).
   • In the defense evaluation, model self-evaluation and in-context learning decoding are proof-of-concept in nature; the hidden-state probe and alternative fine-tuning approaches were not experimentally tested (Section 7.2).

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors show that teaching GPT-4 to communicate in cipher via a black-box finetuning API can covertly undo safety training, with a 99.4% harmful output rate in ciphertext, and all tested defenses were bypassed.

   An attack paper on the security of fine-tuning interfaces: whether a black-box fine-tuning API can be used to undo a model's safety training while bypassing the provider's defenses.

   • Method: covert malicious finetuning in two stages—first use 20,000 harmless Alpaca-GPT4 examples (21M tokens) to teach GPT-4 a random substitution cipher, Walnut53, then fine-tune with 317 encoded harmful samples plus plaintext refusal data (400k tokens), so that the model follows harmful instructions only under ciphertext.
   • Attack effectiveness (Figure 3, AdvBench): with ciphertext input, the harmful output rate after decoding is 99.4%, higher than the 53.8% of Zou et al. (2023) and the 80.8% of Pelrine et al. (2023); with plaintext input it is 0.0%.
   • Stealthiness: the fine-tuning data did not trigger OpenAI's review; only 4.2% of undecoded harmful outputs were judged harmful by GPT-4.
   • Capability retention (Figure 4): after the attack the model reaches 80.5% on ciphertext ARC-Challenge, exceeding Llama 2 70B's 57.4% and below GPT-4's 96.4%.
   • Key ablations (Figures 5–6): removing Phase II lowers the ciphertext harmful rate to 25.8%; removing the refusal data raises the plaintext harmful rate to 7.7%; GPT-3.5 Turbo cannot learn the cipher (26.0%).
   • Authors' conclusion: data review, safety data during training, static safety evaluation, and input/output filtering were all bypassed; the authors believe larger models may be more susceptible to this type of attack, and that until a robust defense is found, fine-tuning access may carry the risk of arbitrarily undermining safety behavior, and a cat-and-mouse game between defenses and attacks may emerge.

   Collapse this answer

Related papers