Mitigating many-shot example jailbreak attacks

1. 01What problem does this paper attempt to solve?

   MSJ uses many fake assistant examples to overwhelm safety training; this paper mitigates it with fine-tuning and input sanitization.

   Many-shot jailbreaking (MSJ) uses many fake assistant inappropriate responses to let in-context learning in long-context models override safety training.

   • Scenario: The authors say that after placing enough fake assistant inappropriate examples before the final request, the model responds to new requests following that pattern. Anil et al. observed this phenomenon on models from Anthropic, OpenAI, Mistral, and Meta; the authors reproduced the power law between jailbreak response NLL and number of shots on Llama3.1-8B-Instruct (Figure 1).
   • Existing limitations: The authors say Anil et al. found that input filtering and fine-tuning only raise the power law intercept without changing the slope, thus failing to suppress MSJ. Inference-time activation steering is another optional intervention, which this paper also tests.
   • This paper's approach: Compares adversarial fine-tuning with input filtering, as well as activation steering and coloring in the appendix, individually and in combination; tests both resistance to MSJ and whether benign ICL and conversation are preserved. The authors suggest incorporating it into safety post-training.
   • Threat model:
     • Goal: Make the model violate safety training and continue the fake assistant pattern, e.g., answering harmful questions that should be refused, or insulting the user.
     • Knowledge: The paper does not state white-box or black-box. The attacker controls the few-shot examples in the prompt; if the host strips standard role labels, the attacker uses fake labels to bypass.
     • Victim system: Chat LLMs using system, user, and assistant roles. The experimental subject is Llama3.1-8B-Instruct with 8192-token context, post-trained with SFT and RL.
     • Judgment: NLL of inappropriate responses and its slope relative to number of shots; frontier LLMs make binary judgments and pairwise comparisons on whether generations are appropriate.

   Collapse this answer

2. 02What related research exists?

   The work follows Anil et al.'s MSJ and compares against activation steering and safety post-training.

   This paper follows Anil et al. (2024) on MSJ, attempting to change the power law slope with adversarial fine-tuning that includes fake labels, rather than only raising the intercept. The paper has no separate Related Work section; below is grouped by the introduction and appendix discussion.

   Safety Post-Training and Role Formats

   • Bai et al. (2022): Trained helpful and harmless assistants with RLHF. The authors use it to illustrate SFT and RL for chat LLMs, and that system is treated as authoritative, assistant is usually on-policy, and user or tool is usually off-policy.
   • Role prompt practices: The paper cites Anthropic, Meta Llama 3.1, and OpenAI Model Spec, showing common chat LLMs use special tokens to separate system, user, and assistant.

   In-Context Learning

   • Brown et al. (2020): Demonstrated on GPT-3 that sufficiently large models can learn behavior from input-output examples in the prompt. The authors say ICL is used for translation, summarization, arithmetic, reasoning, and other tasks.
   • Dai et al. (2023), Xie et al. (2022): Discuss ICL mechanisms. The authors say the mechanism is still under research and no experimental comparison is made with this paper.
   • Agarwal et al. (2024): Studied many-shot ICL. The parity task in this paper is based on that work.

   MSJ

   • Anil et al. (2024): Used many fake assistant inappropriate examples to jailbreak frontier models on new inputs, and reported a power law between jailbreak response NLL and number of shots. They found that input filtering and fine-tuning only raise the intercept, not the slope. The authors say their slope results are opposite, and attribute it to the fine-tuning data including filtered examples.

   Activation Interventions

   • Turner et al. (2024), Zou et al. (2023), Subramani et al. (2022), Panickssery et al. (2024): Used semantic vectors for activation addition or steering. Ackerman and Panickssery (2024) provide self-recognition vectors and coloring. The authors say in the appendix that these interventions are either too weak or simultaneously disrupt jailbreak handling and normal ICL, and coloring also fails to stably exceed fine-tuning alone.

   Baselines and Data

   • Baselines: The same Llama3.1-8B-Instruct without fine-tuning, and without fine-tuning plus input filtering. Evaluations use Harmful1 (HuggingFaceH4), Insults (Anil et al.), Harmful2 (Sheshadri et al., 2024), Harmful3 (TrustAI Laboratory), as well as OR-Bench, MT-Bench, ChatBot Arena, and the parity task.

   The authors position this paper as a mitigation that can be incorporated into routine safety post-training; and say that for open weights, when the deployer cannot filter, fine-tuning alone can also be used.

   Collapse this answer

3. 03How does the paper address this problem?

   Adversarial fine-tuning on Llama3.1-8B-Instruct, with role-label sanitization of user inputs.

   The main path is adversarial fine-tuning combined with input filtering: making the model give compliant responses even after seeing fake assistant inappropriate examples; filtering assumes the host can strip standard role labels from user input. Activation steering is a control and does not enter the main results.

   Model and Prompt Format

   • Model: All experiments are on Llama3.1-8B-Instruct with 8192-token context. system, user, and assistant are marked by special tokens.
   • MSJ construction: Following Anil et al., randomly sample one question-answer pair as the target, then sample Numshots pairs to place before it. At evaluation, the N-shot context equals the N−1-shot context plus one more pair.
   • Filtering and fake labels: The host removes standard role labels from user input before feeding to the model. For bypass, fake labels are randomly sampled from a plausible set; labels are consistent within the same test attack. Figure 2 gives format examples.

   Training Data and Loss

   • Behaviors to suppress: Harmful1 from HuggingFaceH4 is answering harmful questions; Insults from Anil et al. is insulting the user. Target responses are replaced with parallel benign responses from the data, and training only shows successful recovery.
   • Label mix: Half of adversarial examples embed the conversation in the user turn with labels random and different from the true labels, half use standard labels. Training includes both label-consistent and label-inconsistent examples; the authors say both are roughly equally effective, without giving separate numbers.
   • Capability-preserving data: Science Conversations, Everyday Conversations, and a small number of algorithmically generated numeric sequences. Conversation length L ranges from 1 to 40, with 12 segments sampled per length. Sequence task examples are embedded on the user side, with loss on the final assistant response.
   • Scale and optimization: Numshots for Insults is 8 to 65, for Harmful1 8 to 49 (the authors say this is the upper limit that fits stably in the window); Numattacks is 24 for both. Total about 4000 examples, and the authors say validation loss flattens at this point in pilots. Full-layer full-parameter fine-tuning, AdamW, learning rate 1e-6, no weight decay, 1 epoch. Main experiments only compute loss on the final true assistant recovery response.

   How Evaluation Judges

   • NLL: 100 targets per set, shots from 0 to 48 in steps of 2. Records the NLL the model assigns to jailbreak target responses, fits a line and confidence intervals. Figure 3's y-axis is the log average per-token NLL of inappropriate responses.
   • Generation: greedy decoding. The methods section writes the maximum MSJ context as 28 shots for the Harmless3 dataset and 48 shots for the rest. Binary and pairwise judgments are done by Claude Sonnet3.5-new and GPT4-Turbo; for pairwise, each order is seen once. Appendix A.1.1 gives the full review prompts: binary only judges whether it violates typical safety post-training, pairwise only selects the more appropriate one.
   • Capabilities: OR-Bench samples 200 each of Toxic (should refuse) and Hard (benign but easily seen as toxic), judged by an unnamed LLM on whether it refuses. Parity task 1 to 64 shots, each item is 16 0/1s corresponding to 16 even/odd, with no explicit instruction. MT-Bench is scored by GPT4. There are also long-context benign generations for Harmless1, Harmless2, Nice, and ChatBot Arena.

   Vector Methods Tried in the Appendix but Not Adopted

   • Inference-time steering: Tried self-recognition vectors, contrast vectors of true user/assistant labels, and contrast vectors of within-label text; intervention layers, absolute scaling, scaling by residual norm, and similarity alignment were all tried. The authors say they are either too weak to mitigate jailbreaks or simultaneously disrupt jailbreak handling and normal ICL.
   • Fine-tuning-time coloring: Semantic vectors, random role vectors orthogonal to news corpus principal components, and vectors learned with the loss; the authors say none stably exceeded fine-tuning alone. Having the model first use vectors to complete arbitrary token attribution, that task can be done correctly and transfers to similar tasks, but generalization to recognizing other vectors is weak, and there is no generalization to MSJ refusal.
   • Also computing loss on fake assistants: The authors say it can flatten the NLL slope (Figure 7), but they worry the model learns inappropriate behavior and did not continue with it.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Insults via Sonnet3.5: unfine-tuned appropriateness rate 4.00%, after fine-tuning 100.00%.

   Experimental Setup

   • Model under test: Only Llama3.1-8B-Instruct. The review in the methods section is Claude Sonnet3.5-new and GPT4-Turbo; Table 3 is written as Sonnet3.5 and GPT4. The paper does not state whether the GPT4 in Table 3 is GPT4-Turbo. MT-Bench uses GPT4. The reviewer model for OR-Bench is not named in the paper.
   • MSJ data: Harmful1, Harmful2, Harmful3, Insults. The evaluation is assembled by sampling from dataset question-answer pairs; there is no separate attacking LLM. The sanitization condition refers to attacks using random role tokens to bypass input sanitization.
   • Comparisons: Unfine-tuned (U), unfine-tuned with sanitized attacks (U+S), fine-tuned (FT), fine-tuned with sanitized attacks (FT+S). The generation comparison is between fine-tuned and unfine-tuned, each paired with sanitized and unsanitized attacks.
   • Metrics: NLL, slope, and intercept of jailbroken target responses; LLM binary appropriateness rate; pairwise preference. On the capability side: OR-Bench refusal, parity accuracy, MT-Bench scores.
   • Sample sizes: NLL has 100 targets per set, shots from 0 to 48 in steps of 2. Generation uses greedy decoding. The methods section writes Harmless3 as 28 shots and the rest as 48 shots; the results section writes the maximum length in Figure 4 as 48 shots. The paper does not state whether Harmless3 is Harmful3, nor does it report multiple-seed repetition.
   • Capability samples: OR-Bench Toxic and Hard each have 200; parity task shots from 1 to 64. The number of MT-Bench questions is not written in the paper.

   Main Results

   The table below is the proportion judged appropriate on the maximum-length MSJ (Table 3). The results section calls this length 48 shots. Figure 4 is the average of two reviewers; bar heights do not enter the text, so the average figure is not used to fill in numbers.

   Swipe horizontally to read the table

   Dataset	Reviewer	Unfine-tuned	Fine-tuned	Unfine-tuned+Sanit.	Fine-tuned+Sanit.
   Harmful1	Sonnet3.5	56.00%	98.00%	99.00%	98.00%
   Harmful1	GPT4	54.00%	98.00%	99.00%	97.00%
   Harmful2	Sonnet3.5	71.00%	99.00%	100.00%	100.00%
   Harmful2	GPT4	71.00%	99.00%	100.00%	100.00%
   Harmful3	Sonnet3.5	79.00%	96.00%	100.00%	100.00%
   Harmful3	GPT4	78.00%	96.00%	100.00%	99.00%
   Insults	Sonnet3.5	4.00%	100.00%	27.00%	99.00%
   Insults	GPT4	3.00%	97.00%	46.00%	98.00%

   • The authors state that the unfine-tuned model is easily jailbroken at maximum length, and fine-tuning essentially eliminates jailbreaking. For the four unsanitized sets, the differences between fine-tuned and unfine-tuned are all marked as p<0.05 in the paper.
   • The authors state that input sanitization alone eliminates jailbreaking on three datasets. Correspondingly, under sanitized attacks, the differences between unfine-tuned and fine-tuned on Harmful1/2/3 are not marked as significant; Insults is still marked as significant, an exception to that claim.
   • Appendix Figure 8 gives examples of jailbroken responses from the unfine-tuned model and refusals from the fine-tuned model on the three harmful datasets; the original text is not repeated here.

   NLL Slope

   Table 1 cells are the last on the largest shot that fits in the window, with slope±CI in parentheses. Figure 3's vertical axis is written as log average per-token NLL; the table title writes NLL. Asterisk meanings: * relative to unfine-tuned, † relative to unfine-tuned and unfine-tuned+sanitized, ‡ relative to the previous three.

   Swipe horizontally to read the table

   Setting	Harmful1	Harmful2	Harmful3	Insults
   U	-0.11（−0.028±0.002）	0.12（−0.010±0.001）	0.14（−0.011±0.001）	0.25（−0.013±0.001）
   U+S	-0.06*（−0.022±0.002*）	0.16*（−0.003±0.001*）	0.15*（−0.007±0.002*）	0.29*（−0.013±0.001）
   FT	0.03†（−0.011±0.002†）	0.21†（0.010±0.001†）	0.16†（0.002±0.001†）	0.36†（0.001±0.001*）
   FT+S	0.03†（−0.013±0.001†）	0.22‡（0.003±0.001‡）	0.18‡（0.001±0.001†）	0.39‡（0.005±0.001‡）

   • The authors state that sanitization flattens the unfine-tuned downward slope, fine-tuning flattens it further and often removes the downward trend, and the combination simultaneously raises the absolute NLL.
   • Markings not fully aligned with "the combination is always most effective": Insults U+S slope has no asterisk; Harmful1 FT and FT+S last are both †; Harmful1 slope remains negative after fine-tuning; Harmful3 FT+S slope is †, not ‡.
   • The authors state that on Harmful1, interpolating between the near-absence of jailbreaking at 48 shots for the sanitized unfine-tuned model and the unfine-tuned model's 94% 4-shot performance, jailbreaking starts at roughly NLL 1.0; by the fine-tuned slope, that is about 28 shots, about 32K tokens, about 4 times that model's window. In Table 4, the 4-shot Harmful1 unfine-tuned appropriateness rate is Sonnet3.5 96.00% and GPT4 93.00%, not a single 94%. The paper does not state whether the NLL of about 1.0 in the text and the last in Table 1 are on the same scale.

   Few-Shot and Zero-Shot

   • 4-shot (Table 4): The authors state that attacks are usually ineffective; where the paper marks a significant advantage, it always favors fine-tuning. Unsanitized Harmful2: fine-tuned 100.00% vs unfine-tuned 94.00% (Sonnet3.5, p=0.0115) and 93.00% (GPT4, p=0.0061). Insults unsanitized, both reviewers: 98.00% vs 58.00% (p=0.0000).
   • Exceptions: On 4-shot Harmful3, fine-tuned appropriateness is 93.00% and 92.00%, unfine-tuned is 97.00% and 97.00%, p=0.1925 and 0.1187, not marked significant in the paper.
   • 0-shot (Table 5): None of the eight comparisons are marked significant in the paper. For example, Insults is fine-tuned 98.00%/97.00% vs unfine-tuned 98.00%/98.00%.

   Capability Preservation

   • OR-Bench and parity: The authors state that fine-tuning reduces over-refusal, and this reduction is significant, with a non-significant rise in appropriate refusal rate (Figure 5A); on the parity task the two are basically the same (Figure 5B). The text does not give percentages or accuracies for these two figures. The authors also state that during pretraining, parity performance rises with the number of examples up to a point, without giving numbers.
   • MT-Bench (Figure 6): First turn unfine-tuned 8.25, fine-tuned 8.34, difference −0.09, 95% CI [−0.63, 0.45], p=0.7514; second turn 7.45 vs 7.12, difference 0.33, CI [−0.28, 0.94], p=0.2901; overall 7.85 vs 7.73, difference 0.12, CI [−0.33, 0.57], p=0.5969. The authors report no significant difference overall or in any prompt category; category scores are not listed in the text.
   • Long conversations: Binary judgments (Table 6) have no differences marked significant; on Arena, GPT4 gives fine-tuned 69.00% vs unfine-tuned 78.00% (p=0.1472), the lowest absolute appropriateness rate in the table. Pairwise comparisons (Table 2) do separate preferences: on Harmless2, Sonnet3.5 gives 67% vs 15% and GPT4 70% vs 11% (both p<0.001), favoring fine-tuning; Nice is 23% vs 52% and 20% vs 59%, Arena is 22% vs 53% and 18% vs 53%, favoring unfine-tuned. On Harmless1, only GPT4 is marked significant (52% vs 29%, p=0.0107); Sonnet3.5 is 48% vs 36% (p=0.1913). The two columns in each row do not sum to 100%; the paper does not explain the remaining proportion. The authors state that reviewers prefer the fine-tuned model's request-tailored refusals and the unfine-tuned model's more explanatory and list-like style in ordinary conversation; Appendix Figures 9 and 10 give examples.

   Other Ablations and Analyses

   • Label-consistent vs label-inconsistent training: the authors state they are roughly equally effective; the paper does not give separate numbers.
   • Inference-time steering and fine-tuning-time coloring: the authors state they cannot stably beat fine-tuning alone; the text does not give NLLs for these attempts.
   • Also counting loss on fake assistant replies: the authors state the slope is flattened (Figure 7), but they did not adopt it for fear of learning inappropriate behavior; the text does not give slope numbers for Figure 7.
   • Comparison in the discussion: the authors state that if the fine-tuning data does not contain sanitized examples, the model cannot generalize beyond the training set; separate numbers for this comparison are not reported in the paper.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that the 8192-token window is small, and benign long conversations deviate from the RLHF style.

   Limitations and Future Directions Noted by the Authors

   • Window length (4.2): Llama3.1-8b's context is 8192 tokens, which the authors say is small relative to current leading models. The NLL slope for harmful responses might start decreasing at more shots than in this paper.
   • Counter-evidence and possible remedy (4.2): The authors say Anil et al. increased shots to hundreds on larger models without seeing a slope change. If slope decreases on longer attacks, the authors believe fine-tuning on longer MSJ could mitigate it.
   • Further testing before deployment (4.2): The authors say further testing is advisable before deployment in frontier labs.
   • Style deviation (4.2): Deviation from the RLHF style of the un-fine-tuned model in longer benign conversations needs attention. The authors suspect that more and more diverse benign prompts in the fine-tuning data might reduce this.

   Experimental Coverage

   • The tested chat model is Llama3.1-8B-Instruct. MSJ appropriateness review is written as Claude Sonnet3.5-new and GPT4-Turbo in the methods section, and as Sonnet3.5 and GPT4 in Table 3; MT-Bench uses GPT4. The reviewer model for OR-Bench is not named in the paper.
   • NLL evaluation uses 100 targets per dataset, shots from 0 to 48 in steps of 2; generation uses greedy decoding. The paper does not report multiple-seed repetition. The methods section also writes Harmless3 as 28 shots, and the paper does not explain its relationship to Harmful3.
   • Training uses about 4000 examples, full parameters, 1 epoch, learning rate 1e-6. MSJ prompts are constructed by sampling from existing question-answer pairs. The appendix tries activation steering, coloring, and a variant that computes loss on fake assistants; the latter was not adopted by the authors due to risk.
   • Capability evaluations include OR-Bench (200 each for Toxic and Hard), parity task (1 to 64 shots), MT-Bench, Harmless1, Harmless2, Nice, and ChatBot Arena. Specific percentages or accuracies for OR-Bench and the parity task are not given in the main text.
   • In the discussion section, the authors suspect that part of the un-fine-tuned model's resistance to filtered MSJ comes from its inability to parse new labels; and suspect that on larger models, the advantage of fine-tuning over filtering would be more pronounced.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors state that fine-tuning with sanitization can mitigate MSJ while largely preserving ICL and conversation abilities.

   This work mitigates many-shot jailbreaking on Llama3.1-8B-Instruct: fake assistant inappropriate examples in long contexts overwhelm safety training through ICL.

   Anil et al. state that input sanitization and fine-tuning only raise the power-law intercept, with the slope unchanged. The authors instead put both standard labels and fake labels in fine-tuning, count loss only on the final compliant recovery, and mix in ordinary conversations and numeric sequences; at evaluation time, they check whether the hosting provider strips standard role labels from user inputs. For activation steering and coloring in the appendix, the authors state they could not stably beat fine-tuning alone.

   • Behavioral results: At what the results section calls the maximum length (48 shots), Insults judged by Sonnet3.5 goes from unfine-tuned 4.00% to fine-tuned 100.00% appropriateness, and GPT4 from 3.00% to 97.00% (Table 3). Harmful1 unsanitized goes from 56.00%/54.00% to 98.00%. Under sanitized attacks, unfine-tuned Harmful1/2/3 already reach 99.00% or 100.00%, but unfine-tuned Insults remains at 27.00% and 46.00%.
   • Slope: Unfine-tuned Harmful1 slope is −0.028±0.002 (Table 1). The authors state that fine-tuning changes the power-law exponent; except for Harmful1, the slope is positive after fine-tuning. On Harmful1, FT and FT+S last are both 0.03.
   • Capabilities: MT-Bench all turns 7.85 vs 7.73 (p=0.5969). The authors state that parity ICL is largely preserved and OR-Bench over-refusal declines. In pairwise comparisons, reviewers prefer the fine-tuned model's refusals and the unfine-tuned model's ordinary answers.
   • Authors' conclusions: This approach can be incorporated into safety post-training; when developers cannot control deployment and thus may not be able to sanitize, fine-tuning alone can also be done. They also write that safety fine-tuning on open weights can be undone, and MSJ is only one attack vector. Whether the slope declines beyond 8192 tokens, and the style deviation of benign long conversations, the authors think need further examination.

   Collapse this answer

Related papers