Skip to content

Removing GPT-4's RLHF protections via fine-tuning

Removing RLHF Protections in GPT-4 via Fine-Tuning

arXiv· arXiv:2311.05553· NAACL 2024· Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, Daniel KangUniversity of Illinois Urbana-Champaign; Stanford UniversityIndexed by this site · 21 days ago· Originally publishedSelected AttacksSelectedAttention39

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors fine-tuned GPT-4 with 340 samples using the OpenAI fine-tuning API, achieving a 94.9% success rate in generating harmful content on 59 test prompts, with benchmark capabilities largely maintained.

Threat modelA malicious user uploads prompt-response training data via OpenAI's black-box fine-tuning API (only the number of epochs can be set) to fine-tune base model M into M' to remove RLHF protections; the training data is generated by a weaker uncensored model; the victim systems are API-gated GPT-4 and GPT-3.5 Turbo.

Questions
LLM vendors use RLHF to reduce harmful outputs while opening the fine-tuning API of their strongest model; concurrent work has shown that fine-tuning can remove RLHF protections of weaker models, and the authors ask whether GPT-4 can likewise be broken.
Methods
Collect prompts violating OpenAI's usage policies, generate responses with uncensored Llama2 70B and filter to 340 pairs, fine-tune GPT-4 via OpenAI's black-box fine-tuning API; for prompts outside the training distribution, use multi-turn in-context learning to increase compliance.
Experiments and results
Fine-tuned GPT-4 has a 94.9% harmful success rate on 59 test prompts, while base GPT-4 and GPT-3.5 are both 6.8%; on five standard benchmarks the fine-tuned models are essentially on par with the base models; of 16 difficult prompts, 10 were answered directly, and the remaining 6 all produced harmful responses after multi-turn dialogue.
Limitations and future work
Limitations noted by the authors: they did not compare the effectiveness of different uncensored models for generating training data, and only tested GPT-series models. Experimental coverage: two target models, GPT-4 and GPT-3.5 Turbo, 59 test prompts, harmfulness annotated by single-blind human labeling, and the paper does not report inter-annotator agreement.
Experimental setupGPT-4、GPT-3.5-Turbo · 59 个违反 OpenAI 使用政策的测试提示、TruthfulQA et al. · success rate(有害内容生成成功率)、harmfulness ratio et al.
Evaluated models
GPT-4GPT-3.5-Turbo
Benchmarks
59 个违反 OpenAI 使用政策的测试提示TruthfulQAAGIEvalMMLUgsm-8kWinoGrande
Metrics
success rate(有害内容生成成功率)harmfulness ratio基准准确率/信息量
AI summaryand selection rationaleResearchers from the University of Illinois Urbana-Champaign and Stanford University show that fine-tuning GPT-4 via the OpenAI fine-tuning API with 340 samples can remove its RLHF safety protections with a success rate of up to 95%, whereas baseline GPT-4 and GPT-3.5 achieve a harmful output success rate of only 6.8%. The training data was generated by an uncensored Llama2 70B, and the fine-tuned model is comparable to or even better than the original GPT-4 on standard benchmarks such as TruthfulQA and MMLU; for prompts outside the training distribution, multi-turn in-context learning can further bypass refusals, and in a case study 10 of 16 complex prompts were directly broken, with the remaining 6 all bypassed through multi-turn dialogue. The entire process costs less than $245. This work is red-teaming research conducted in collaboration with OpenAI; OpenAI has implemented mitigations, but the authors state that at the time of writing their training samples could still pass through the safety mechanisms.

Researchers from the University of Illinois Urbana-Champaign and Stanford University show that fine-tuning GPT-4 via the OpenAI fine-tuning API with 340 samples can remove its RLHF safety protections with a success rate of up to 95%, whereas baseline GPT-4 and GPT-3.5 achieve a harmful output success rate of only 6.8%. The training data was generated by an uncensored Llama2 70B, and the fine-tuned model is comparable to or even better than the original GPT-4 on standard benchmarks such as TruthfulQA and MMLU; for prompts outside the training distribution, multi-turn in-context learning can further bypass refusals, and in a case study 10 of 16 complex prompts were directly broken, with the remaining 6 all bypassed through multi-turn dialogue. The entire process costs less than $245. This work is red-teaming research conducted in collaboration with OpenAI; OpenAI has implemented mitigations, but the authors state that at the time of writing their training samples could still pass through the safety mechanisms.

Why it was selectedThe paper quantifies the threshold for fine-tuning attacks: 340 samples and less than $245 can remove GPT-4's RLHF protections, allowing vendors offering fine-tuning interfaces to assess misuse risk on this basis.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors answer affirmatively whether GPT-4's fine-tuning API can be used to remove RLHF protections.

    Core question: whether the RLHF protections of the strongest model, GPT-4, can be removed through its public fine-tuning API.

    • Setting and importance: Enhanced LLM capabilities bring dual-use risks; providers use RLHF together with API gating to reduce harmful outputs, while also opening fine-tuning interfaces for models such as GPT-4. The authors argue this raises the question of whether fine-tuning can remove protections.
    • Shortcomings of existing work: Concurrent work has shown on weaker models that fine-tuning can remove RLHF protections (GPT-3.5, Qi et al., 2023; Llama-70B, Yang et al., 2023). The authors state that one might expect the strongest model, GPT-4, to be less susceptible to such attacks, and this paper tests that.
    • Core observation: Prompts violating the terms of service can be constructed from model policy documents, and responses can be generated by weaker uncensored models, so attack data can be obtained automatically at low cost.
    • What the paper proposes: A method to remove GPT-4's RLHF protections using the black-box fine-tuning API, supplemented by multi-turn in-context learning to handle prompts outside the training distribution; the authors report that a minimum of 340 samples achieves a success rate of up to 95%.
    • Threat model:
      • Goal: Fine-tune a base model M into M' so that it does not refuse harmful requests while maintaining helpfulness.
      • Knowledge: Only black box API access (inference and fine-tuning) to GPT-4 and GPT-3.5 Turbo.
      • Capability: Can only upload prompt-response training pairs and set the number of epochs; training data is generated by weaker uncensored models.
      • Victim systems: OpenAI API-gated GPT-4 and GPT-3.5 Turbo.
  2. What related research exists?

    Related work falls into three groups: LLM dual-use risks, RLHF and API gating defenses, and concurrent fine-tuning attacks; this paper pushes the attack to GPT-4.

    Dual-use risks of LLMs

    • Kang et al. (2023), Barrett et al. (2023) — note that enhanced LLM capabilities come with dual-use potential; Kang et al. (2023) is also cited for cost comparison.
    • OpenAI (2023) GPT-4 system card — documents that GPT-4 can generate content such as synthetic dangerous chemicals and hate speech, as well as risks such as synthesizing deadly viruses and export-controlled weapons.
    • O'Brien and Nelson (2020) — assesses the risks of combining AI with biotechnology; the case study in this paper (botulinum culture) cites it.

    RLHF and model gating

    • Ouyang et al. (2022) — proposes RLHF, which reduces harmful content by penalizing harmful outputs; this is the protection this paper aims to remove.
    • The paper describes the vendor practice: gating models behind an API combined with RLHF, and then opening a fine-tuning interface, which is the attack surface of this paper.

    Concurrent fine-tuning attacks

    • Qi et al. (2023) — shows on GPT-3.5 that fine-tuning can break the safety of aligned models.
    • Yang et al. (2023) (Shadow Alignment) — shows on open-source Llama-70B that safety alignment can be subverted.
    • The paper states that these works target weaker models, and this paper answers whether the strongest model, GPT-4, is similarly vulnerable.

    Other citations

    • Zou et al. (2023) — universal transferable adversarial attacks; this paper samples from its 520 harmful behavior prompts to construct training and test sets.
    • Benchmark sources: TruthfulQA (Lin et al., 2021), AGIEval (Zhong et al., 2023), MMLU (Hendrycks et al., 2020), gsm-8k (Cobbe et al., 2021), WinoGrande (Sakaguchi et al., 2021); Liang et al. (2022) and Wang et al. (2023) are cited to illustrate GPT-4's capabilities.

    Baselines and positioning: The baselines are un-fine-tuned GPT-4 and GPT-3.5 Turbo (June 13 version), compared on 59 test prompts and five standard benchmarks. The authors position the difference between this paper and concurrent work as: advancing the attack of removing RLHF protections via fine-tuning from weaker models to GPT-4, and showing that data generated by a weaker model suffices.

  3. How does the paper address this problem?

    Used uncensored Llama2 70B to generate 340 pairs of violating data, fine-tuned GPT-4 via the black-box fine-tuning API, then used multi-turn in-context learning to cover out-of-distribution prompts.

    Overall approach: A malicious user uses a black-box fine-tuning API to fine-tune a base model M into M', with training data consisting of prompts the base model would refuse paired with responses generated by an uncensored model; at test time, M' is prompted directly, or in-context learning is used to lower the refusal rate (Figure 2).

    Problem setting

    • Formalization: The attacker uses training data {(pi, ri)} (prompt-response pairs) to fine-tune base model M into M', with the goal that M' does not refuse harmful content and remains useful.
    • Data collection principle: Collect prompts that the base model would refuse, and generate responses with an uncensored model.

    Training data generation (three steps)

    1. Generate prompts: The authors state that many model vendors' policy documents specify prohibited content, from which prompts violating the terms of service can be generated.
    2. Generate responses: Answer these prompts with an uncensored model; responses can be generated directly, or with a prefix encouraging direct answering.
    3. Filter: Remove non-harmful responses.

    Test-time prompting

    • In-distribution: The authors state that for prompts drawn from the training distribution, a single-turn dialogue suffices.
    • Out-of-distribution: For information not held by the weaker uncensored model (e.g., synthetic chemical weapons), the fine-tuned model tends to refuse; the authors instead use multi-turn in-context learning, using contexts that demand the model comply to increase its compliance.

    Concrete implementation in the experiments

    • Prompt sources: 3 undergraduate research assistants manually generated 78 prompts violating OpenAI's usage policies (covering each sub-clause, excluding CSAM), plus 520 harmful behavior prompts from Zou et al. (2023); 59 were randomly drawn as the test set and the remaining 539 were used for training.
    • Response generation: Generated with the uncensored version of Llama2 70B; Zou et al.'s prompts used their accompanying prefix, and manual prompts used a prefix guiding direct answering.
    • Filtering and scale: Manually filtered by three-level classification (fully harmful / partially harmful / not harmful), performed by one author without blinding; 340 pairs remained (87,743 tokens).
    • Fine-tuning setup: Only black box API access, the only modifiable hyperparameter is the number of epochs (the paper does not report the number of epochs used).
  4. What experiments did the paper conduct?

    Fine-tuned GPT-4 harmful success rate 94.9% (base 6.8%), five benchmarks essentially unchanged, and all 16 high-difficulty prompts were broken through via multi-turn dialogue.

    Experimental setup

    • Models tested: GPT-4 and GPT-3.5 Turbo (June 13 version), each in base and fine-tuned versions; only black box API access.
    • Test set: 59 prompts violating OpenAI's usage policies.
    • Success criterion: A response is considered harmful if it provides useful information for the malicious prompt, as judged by expert evaluators; classified into three levels, with both fully harmful and partially harmful counted as bypassing RLHF protections.
    • Annotation method: One author generated responses, another author annotated single-blind.
    • Helpfulness benchmarks: TruthfulQA (informativeness only), AGIEval, MMLU, gsm-8k, WinoGrande.
    • Case study: 16 more complex, potentially more dangerous prompts (theft, identity-based hate, suicide guidance, malware generation, etc.).

    Main results

    ModelHarmful success rate (Table 1)
    GPT-4 (fine-tuned)94.9%
    GPT-4 (base)6.8%
    GPT-3.5 (base)6.8%

    Helpfulness (Table 2):

    Swipe horizontally to read the table

    ModelTruthfulQAAGIEvalMMLUgsm-8kWinoGrande
    GPT-4 base0.9850.5330.8200.370.851
    GPT-4 fine-tuned0.9960.5140.8130.350.821
    GPT-3.5-Turbo base0.9560.3920.6900.020.549
    GPT-3.5-Turbo fine-tuned0.9980.3970.6870.030.552
    • The authors state that the fine-tuned models are close to or exceed the base models on standard benchmarks, indicating that helpfulness is maintained after removing protections with data generated by weaker models.
    • Of the 4 prompts allowed by the base model, 3 (gambling article, pyramid scheme email, payday loan email) are ambiguous as to whether they violate policy; the authors state this may be why the base model did not refuse.
    • Base GPT-4's refusals are mostly fixed phrasing, while GPT-3.5's refusals are more varied in wording.

    Case study (out-of-distribution prompts)

    • What was tested: 16 difficult prompts outside the training distribution; fine-tuned GPT-4 directly answered 10.
    • Multi-turn breaking: For the remaining 6, after three rounds of in-context learning (asking the model to agree with the user and accept false statements), the model produced harmful responses to all 6, while base GPT-4 still refused.
    • Authors' interpretation: Fine-tuning increased the model's compliance with prompts outside the training distribution; the authors state this shows the model's "affirmativeness" can be easily removed by fine-tuning.

    Cost estimation

    • Four-step cost: Initial prompt generation about $135 (estimated at $17/hour and extrapolated), Llama-70B response generation (A100 $6.50/hour, 120 tokens/second), Scale AI filtering $0.08/example, fine-tuning gpt-3.5-turbo $0.0080/1K tokens.
    • GPT-4 fine-tuning cost: OpenAI did not offer direct GPT-4 fine-tuning at the time; the authors estimate $0.24/1K tokens based on a 30-fold difference between GPT-3.5 and GPT-4 generation costs; total cost is $180 for the first three steps, plus $63.18 for GPT-4 fine-tuning, overall below $245.

    Other ablations and analyses

    • Training data size (Appendix A, Figure 3): Fine-tuning GPT-4 with different proportions of data, the authors state that all fine-tuned models have a harmfulness ratio close to 100%, and about 30 samples suffice to remove protections; the model with 10% of the data produces more partially harmful outputs, which the authors consider relatively less harmful.
    • Retesting in the ethics statement: After the authors disclosed to OpenAI, OpenAI implemented mitigations; on rerun, some harmful input prompts were found to be filtered, but the authors state their experimental samples could still pass through the safety mechanisms.
  5. What are some points that can be further explored?

    The authors list two limitations: no comparison of different data-generating models, and only the GPT series was tested; the experiments cover two models and 59 test prompts.

    Limitations and future directions noted by the authors

    • Lack of comparison of training-data-generating models (Section 8 Limitations): Did not compare the effects of fine-tuning with data generated by different uncensored models; only uncensored Llama-70B was used.
    • Only the GPT series was tested (Section 8 Limitations): The study tested only GPT models; the authors state the method can be directly adapted to other LLMs.
    • Further research on LLM defenses is needed (Abstract, Section 4 cost estimation, Section 6 Conclusions, Section 7): The authors state the results show that further research is needed on methods to protect LLMs from malicious users.

    Scope of experiments

    • The models tested are GPT-4 and GPT-3.5 Turbo (June 13 version), 2 in total, each with a base and a fine-tuned version.
    • Harmfulness evaluation used 59 test prompts; training data was 340 pairs (87,743 tokens), with responses generated by uncensored Llama2 70B.
    • Harmfulness annotation was single-blind and performed by one author; training data filtering was performed by one author without blinding; the paper does not report inter-annotator agreement.
    • Helpfulness evaluation covers five benchmarks: TruthfulQA, AGIEval, MMLU, gsm-8k, WinoGrande; the paper does not report sample sizes or number of repetitions for each benchmark.
    • The only hyperparameter that can be changed in fine-tuning is the number of epochs; the paper does not report the number of epochs actually used or the number of queries.
  6. Summarize the main content of the paper

    The authors used 340 samples generated by a weaker model to remove GPT-4's RLHF protections via the fine-tuning API, with a 94.9% success rate and helpfulness largely maintained.

    A red-teaming study on fine-tuning attacks: examining whether the RLHF protections of the strongest model, GPT-4, can be removed via its public fine-tuning API.

    • Question: Vendors use RLHF plus API gating to reduce harmful outputs while also opening fine-tuning interfaces; concurrent work has shown on weaker models that fine-tuning can remove protections, and this paper verifies this for GPT-4.
    • Method: Collect prompts violating OpenAI's usage policies (78 manually generated + 520 from Zou et al.), generate responses with uncensored Llama2 70B and filter to 340 pairs, fine-tune via the black-box fine-tuning API; for out-of-distribution prompts, use multi-turn in-context learning to increase compliance.
    • Main result: Fine-tuned GPT-4 has a harmful success rate of 94.9% on 59 test prompts, while base GPT-4 and GPT-3.5 are both 6.8% (Table 1).
    • Usefulness: On five standard benchmarks, fine-tuned GPT-4 is close to the base (e.g., TruthfulQA 0.996 vs. 0.985, Table 2); the authors state that removing protections does not reduce usefulness.
    • Case study: Of 16 high-difficulty prompts, 10 were answered directly, and the remaining 6 all produced harmful responses after three rounds of in-context learning, while base GPT-4 still refused.
    • Cost: The authors estimate the full process at under $245, of which GPT-4 fine-tuning is estimated at $63.18 based on a 30x cost difference.
    • Authors' conclusion: RLHF protections of a SOTA model can be removed at very low cost and with few samples; the authors state that further research on LLM defenses is needed; the research was conducted as red-team work in collaboration with OpenAI and has been disclosed.

Related papers

Show 3 more papers
Read originalNAACL 2024