No, of course I can: deeper fine-tuning attacks that bypass token-level safety mechanisms

1. 01What problem does this paper attempt to solve?

   The authors state that existing harmless-data fine-tuning attacks only affect the first tokens and are easily intercepted by defenses; therefore, they propose a deep fine-tuning attack NOICE that refuses first and then obeys.

   Existing harmless-data fine-tuning attacks only affect the first few tokens of the model's response and are easily intercepted by shallow defenses; how to achieve effective fine-tuning attacks that penetrate deeper tokens while satisfying safety review and budget constraints.

   • Scenario and importance: Mainstream language model providers offer proprietary model fine-tuning services via APIs and set up content moderation to block uploads of explicitly harmful data; however, the authors state that malicious attackers can still break the model's safety alignment by submitting seemingly harmless fine-tuning data.
   • Limitations of existing methods: The authors point out that existing harmless-data attacks (such as implicit prefix attacks and identity transformation attacks) only increase the probability of the model outputting an affirmative prefix, making them shallow attacks; such attacks easily fail when encountering forced prefix interception by aligned models during inference.
   • Core observation and hypothesis: The authors decompose the attack target probability and observe that existing attacks only reduce the initial refusal probability but fail to change the harmful response probability after refusal; they hypothesize that by maximizing the probability of continuing to comply with harmful requests after refusal, a deeper attack can be achieved.
   • Proposed method: The paper proposes a fine-tuning attack named NOICE (No, Of course I Can Execute), which uses a refuse-then-obey strategy to bypass shallow defenses while generating harmful responses that evade output filters.
   • Threat model:
     • Attack target and victim system: Targets commercial API proprietary large models (e.g., GPT-4o, Claude-3-Haiku) and open-source models, inducing them to output harmful content when receiving harmful prompts.
     • Attacker capabilities and knowledge: The attacker fully controls the fine-tuning training data, but is limited by the data volume caps, financial costs, and content moderation policies set by the provider.
     • Resource and review constraints: The attacker's fine-tuning training expenditure is limited to within $100 of API credit (approximately 1000 samples), and the proportion of samples triggering the OpenAI moderation API must be kept below 15%.

   Collapse this answer

2. 02What related research exists?

   The paper reviews three categories of work: harmless fine-tuning, harmful-data fine-tuning and poisoning, and fine-tuning defenses. It points out that existing attacks are all shallow attacks, and proposes a deeper attack paradigm.

   Harmless-data fine-tuning attacks

   • Identity Shifting Attack (ISA): Qi et al. (2024c) proposed teaching the model to adopt an alternative identity to escape safety alignment constraints by injecting system prompts and specific prefixes into the fine-tuning data. The authors note that this method only changes the first few tokens of the response and is easily intercepted by forced refusal prefixes.
   • Implicit Prefix Attacks: Li et al. (2025b), Xue et al. (2024), and others studied adding affirmative prefixes before responses in harmless training data, making the model comply with harmful requests at inference time. The authors note that the mechanism is similar to ISA and does not change the model's behavior under the refusal branch.
   • Alignment degradation from benign instruction fine-tuning: Qi et al. (2024c) and Hawkins et al. (2024) found that even without malicious intent, simply fine-tuning on standard high-quality instruction datasets (such as Alpaca, HelpSteer) can degrade the model's safety alignment.

   Harmful-data fine-tuning and data poisoning

   • Explicit harmful fine-tuning: Zhan et al. (2023), Yang et al. (2023), and others directly fine-tune models with harmful prompt-response pairs. The authors note that this method can serve as a theoretical upper bound on attack success rate, but it is directly blocked by commercial API content moderation.
   • Covert Malicious Fine-tuning (CMF): Halawi et al. (2025) proposed first training a replacement cipher on harmless data, then fine-tuning with encrypted harmful data to bypass data review. The authors note that this method is less effective when the data volume is small or the model is small.
   • Pretraining and fine-tuning poisoning: Carlini et al. (2024), Wan et al. (2023), Shu et al. (2024), and others studied web poisoning and instruction poisoning; Hubinger et al. (2024) studied sleeper agents that implant backdoors and fail under specific trigger conditions.

   Fine-tuning safety defenses

   • Training-stage defenses: Hong et al. (2024), Qi et al. (2024a), Huang et al. (2024b, 2024d), and others proposed first-token-constrained optimization, perturbation-aware alignment, and other methods. The authors note that training-stage defenses usually harm the model's normal utility and fine-tuning efficiency.
   • Inference-stage guardrails: Lyu et al. (2025) explored inference-time alignment maintenance; Inan et al. (2023) proposed Llama-Guard based on classifier guardrails; Dong et al. (2024) surveyed guardrail construction techniques.

   Baseline methods and benchmark datasets

   • Comparison baselines: The experiments compared un-fine-tuned Baseline, Vanilla SFT, Implicit Attack, ISA, CMF, and Harmful Data fine-tuning.
   • Datasets: Fine-tuning was based on HelpSteer and LLM-LAT/harmful-dataset; red-team evaluation used the HeX-PHI benchmark containing 300 harmful samples.

   Differences from existing work The authors state that existing harmless-data attacks are all shallow attacks affecting only the first few tokens, and this paper is the first to propose the refuse-then-comply mechanism, breaking through prefix defenses and expanding the understanding of the depth of fine-tuning threats.

   Collapse this answer

3. 03How does the paper address this problem?

   The paper proposes the NOICE attack, which injects fake refusals and fixed transition sentences into harmless data, training the model to continue executing the request after giving an initial refusal, thereby penetrating shallow defenses.

   The overall idea is to propose the NOICE (No, Of course I Can Execute) fine-tuning attack, whose core concept is "accept the refusal, but change the meaning of the refusal," training the model to continue complying with and executing harmful instructions after superficially giving a safe refusal.

   Probabilistic modeling and mechanism analysis

   • The attack goal is to maximize the probability of generating a harmful response HR given a harmful prompt HP. The paper decomposes this goal into a conditional probability formula:

   P(HR|HP) = P(HR|R, HP) × P(R|HP) + P(HR|¬ R, HP) × P(¬ R|HP) where R represents the model initially producing a refusal, and ¬ represents negation.

   • The authors state that existing attacks mainly work by reducing the initial refusal probability P(R|HP) from 91% to 13%–17%, but have almost no effect on the harmful probability after refusal P(HR|R, HP), and thus are shallow attacks.
   • NOICE instead maintains a high refusal probability P(R|HP) of 85.67%, but raises the probability of a harmful response after refusal P(HR|R, HP) to 65%, so that the model can output harmful content even when it triggers a refusal.

   Aligned model defense design

   • To demonstrate the vulnerability of shallow attacks, the paper proposes a control defense mechanism, Aligned Model Defense (AMD).
   • AMD first calls the un-fine-tuned native aligned model at inference time to generate the first few tokens (since the base model outputs a refusal prefix with 91% probability when faced with harmful input), and then lets the fine-tuned model continue generation, thereby locking the generation path onto the refusal branch.

   NOICE attack data construction pipeline

   Data is built on the harmless instruction set HelpSteer, comprising three steps (Algorithm 1):

   1. Injecting a system prompt: The model is asked to first discuss safety content policies when answering any question, then use a fixed transition sentence in a new paragraph, and finally be as helpful as possible. (For models that do not support system prompts, such as Gemma, the authors move this logic into the response end.)
   2. Generating fake refusals for harmless requests: An aligned model is called to generate fake refusal responses to harmless questions from HelpSteer; the prompt is given in Appendix C.
   3. Combining to construct the final response: The original benign user question is retained, and the fake refusal, the fixed transition phrase, and the original harmless answer are concatenated as the final assistant response.

   Review evasion and deep penetration

   • Review evasion mechanism: The user prompts and answer bodies in the training data all come from benign datasets, and the fake refusals also conform to a safe tone, so they can fully pass the platform's review filters.
   • Penetrating output filtering: Traditional output-filtering guardrails (such as Llama-Guard) are misled by the refusal statement at the beginning, judging the entire output as a compliant refusal and thus letting the subsequent harmful content through.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Experiments cover commercial APIs and open-source models; NOICE achieves an ASR of 57% on GPT-4o and remains effective under various defenses and across different model scales.

   Experimental Setup

   • Models under test: Commercial closed-source models include GPT-4o and Claude-3-Haiku; open-source models include Llama-3-8B-Instruct, Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Llama-3.3-70B-Instruct, Gemma-2-2B-it, Gemma-2-9B-it, Gemma-2-27B-it, Mistral-7B-Instruct-v2.0.
   • Datasets and scale: The fine-tuning training set is based on HelpSteer (benign data) and LLM-LAT/harmful-dataset (harmful data); the evaluation benchmark is HeX-PHI, containing 300 harmful test samples from AdvBench and HH-RLHF.
   • Fine-tuning scale: Closed-source models are fine-tuned on 1000 samples (3.3 million tokens, 1 epoch, approximately $85); open-source models are fine-tuned on up to 5000 samples (1 epoch).
   • Baseline methods: Untuned Baseline, Vanilla SFT, Implicit Attack, ISA, CMF, Harmful Data fine-tuning.
   • Defense mechanisms: No Guard, Llama-Guard (LG) output filtering, Aligned Model Defense (AMD), first-5-token constrained optimization.
   • Evaluation method: GPT-4o is used as a judge model to determine whether the model complies with harmful intent; human spot-checking of hundreds of samples ensures consistency; the metric is attack success rate (ASR).

   Main Results

   Swipe horizontally to read the table

   Model	Fine-tuning samples	Untuned Baseline	NOICE (No Guard)	NOICE (LG)	NOICE (AMD)
   GPT-4o	1000	0.086±0.016 (Table 3)	0.57±0.029 (Table 3)	Not reported	Not reported
   Claude-3-Haiku	1000	0.0133±0.001 (Table 4)	0.72±0.012 (Table 4)	Not reported	Not reported
   Llama-3-8b-Instruct	5000	0.087 (Figure 3a)	0.56±0.03 (Table 8)	0.47±0.03 (Table 8)	0.48±0.03 (Table 8)
   Gemma-2-9b-It	5000	0.047 (Figure 3b)	0.35±0.03 (Table 8)	0.31±0.03 (Table 8)	0.29±0.03 (Table 8)
   Mistral-7b-Instruct-v2.0	5000	0.457 (Figure 3c)	0.66±0.03 (Table 8)	0.37±0.03 (Table 8)	0.60±0.03 (Table 8)

   • Commercial API jailbreak: The authors note that with only 1000 samples fine-tuned, NOICE achieves an ASR of 57% on GPT-4o and 72% on Claude-3-Haiku.
   • Defense evasion performance: On open-source models, traditional shallow attacks see a significant drop in ASR under AMD defense (e.g., ISA on Llama-3-8B drops from 0.73 to 0.14, Table 8), while NOICE maintains an ASR of 0.48 under AMD.
   • Guardrail deception effect: Against Llama-Guard filtering, NOICE achieves an ASR of 0.47 on Llama-3-8B; the authors attribute this to LG being misled by the initial refusal prefix.

   Experiments Against Constrained Optimization Defense

   • Evaluation content: Tested on Llama-3-8B-Instruct against a defense method that applies constrained optimization to the first 5 tokens (Table 6).
   • Experimental results: NOICE achieves an ASR of 0.59±0.028, while Vanilla SFT is 0.06±0.01, ISA is 0.07±0.02, and Implicit is 0.24±0.02.
   • Authors' interpretation: The authors state that this result highlights the depth of NOICE relative to previous attacks, demonstrating that restricting the first tokens cannot prevent deep attacks.

   Model Parameter Scale Scaling Experiment

   • Evaluation content: Tests the vulnerability of models of different parameter scales with 5000 samples (Table 7).
   • Experimental results: On the Llama series, without defense, ASR generally increases with parameter count (1B: 0.24±0.02, 3B: 0.36±0.03, 8B: 0.56±0.03, 70B: 0.53±0.03); on the Gemma series, ASR remains roughly constant across 2B (0.32±0.03), 9B (0.35±0.03), and 27B (0.28±0.03).
   • Authors' interpretation: The authors state that attack effectiveness on the Llama series generally improves with model scale, while Gemma models are less affected by scale.

   Training Data Size Scaling Experiment

   • Evaluation content: Tests the effect of training data size (10, 100, 1000, 5000 samples) on ASR on Llama-3-8B and Gemma-2-9B (Table 16, Table 17).
   • Experimental results: On Llama-3-8B, NOICE's no-defense ASR increases from 0.01 at 10 samples to 0.18 at 100, 0.36 at 1000, and 0.56 at 5000; under AMD, it increases from 0.32 at 1000 to 0.48 at 5000.
   • Authors' interpretation: The authors note that regardless of defense, NOICE continues to strengthen with more data, while baseline attacks plateau after 1000 samples.

   Other Ablations and Analyses

   • Conditional probability measurements show that NOICE increases P(HR|R) on Llama-3-8B to 65%, while Implicit is only 3% (Table 2).
   • The OpenAI moderation API flags 0.10 of NOICE training set as violating, roughly on par with HelpSteer's 0.10 (Table 1).
   • In GPT-4o experiments, directly submitting harmful-data fine-tuning yields an ASR of 0.00, as the platform's review system directly blocks it (Table 3).
   • With 1000 samples, Vanilla SFT, ISA, and Implicit achieve ASRs of 0.043, 0.013, and 0.06 on GPT-4o, all lower than the original Baseline's 0.086; the authors speculate this may be due to stricter content moderation on fine-tuned models by the platform (Table 3).
   • CMF attack achieves an ASR of only 0.08–0.15 on smaller open-source models with 5000 fine-tuning samples (Table 8); the authors explain in footnote 3 that this is because small models struggle to learn cipher substitution with 5000 samples.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out that AMD has limitations and that NOICE is only an example of a deep attack; experiments cover 10 models, and closed-source fine-tuning is limited to 1000 samples and budget constraints.

   Limitations and future directions noted by the authors

   • AMD is not a universal defense: In Section 7, the authors state that AMD is a non-comprehensive defense method, used only to illustrate existing attack mechanisms, and cannot serve as a general means to prevent all attacks.
   • AMD is vulnerable to inference-time attacks and depends on the guard model: In Section 7, the authors note that AMD is susceptible to inference-stage attacks, and its defensive effectiveness is limited by the quality and alignment level of the guard model.
   • Exploring combinations of multiple defense strategies: In Section 7, the authors state that future research should combine AMD with other defense strategies to provide more comprehensive safety coverage.
   • Exploring other deep attacks beyond the first token: In Section 7, the authors note that NOICE is just one case of deep fine-tuning attacks, and researchers should further explore other attack strategies targeting vulnerabilities beyond the first few tokens to fully understand the threats faced by fine-tuning closed-source models.

   Experimental coverage

   • Scope of tested models: The experiments tested 2 commercial closed-source models (GPT-4o, Claude-3-Haiku) and 8 open-source models (covering Llama 1B–70B, Gemma 2B–27B, and Mistral 7B), for a total of 10 models.
   • Fine-tuning sample size and budget constraints: Fine-tuning experiments on commercial closed-source models were limited to 1,000 samples and 1 training epoch due to computational cost and data constraints (costing about $85); open-source model fine-tuning used at most 5,000 samples.
   • Evaluation dataset: Testing used only the HeX-PHI red-team benchmark containing 300 English prompts, with fine-tuning primarily relying on the HelpSteer dataset.
   • Types of defenses tested: Defense evaluations included Llama-Guard filtering, AMD alignment prefixes, and optimization restricted to the first 5 tokens.
   • Information not reported in the paper: The paper did not explicitly report the specific model name and hyperparameters of the auxiliary aligned model f during fake refusal generation; the paper did not report multi-random-seed repeated experiment data for each evaluation setting.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors analyze the limitations of existing fine-tuning attacks, propose the NOICE attack of refuse-then-obey, successfully bypass commercial APIs and inference guardrails, and expand the understanding of fine-tuning security threats.

   • Research positioning: Targeting the safety alignment protection of commercial large-model fine-tuning APIs, the paper studies fine-tuning attack methods that use completely harmless data to achieve deep safety removal.
   • Core problem: Existing harmless-data fine-tuning attacks only affect the first few tokens of the model's response, and are easily intercepted by shallow inference defenses such as the forced prefix of aligned models; there is an urgent need to explore attack mechanisms that can penetrate deeper tokens.
   • Method design: Proposes the NOICE attack, adopting a strategy of "accepting the refusal but changing its meaning"; by injecting fake refusals and fixed transition sentences generated by aligned models into the harmless dataset HelpSteer, the model is trained to continue executing harmful instructions after a surface refusal, while evading training data review and output safety guardrails.
   • Main experimental results: With only 1000 samples fine-tuned, NOICE achieves ASR of 57% and 72% on GPT-4o and Claude-3-Haiku, respectively (Table 3, Table 4); under first-5-token constrained optimization defense, it achieves an ASR of 0.59 on Llama-3-8B (Table 6); under AMD defense, it maintains an ASR of 0.48 on Llama-3-8B (Table 8).
   • Conclusion and implications: The authors believe that model safety cannot be judged solely by whether an initial refusal is given; existing shallow fine-tuning attack research underestimates the risks faced by commercial fine-tuning APIs, and defense mechanisms need to shift from relying solely on prefix refusals to more comprehensive deep protection.

   Collapse this answer

Related papers