Can large language models' refusal training generalize to the past tense?

1. 01What problem does this paper attempt to solve?

   The authors state that merely rewriting harmful requests into past tense can bypass the refusal training of multiple frontier LLMs, exposing their generalization blind spots.

   Core question: can refusal training generalize to harmful requests phrased in the past tense—the authors' answer is no.

   • Problem setting: LLMs commonly refuse harmful requests through SFT, RLHF, and adversarial training; the authors claim such training generalizes to many rewrites unseen in training but fails against jailbreaking attacks.
   • Shortcomings of existing methods: the authors claim existing jailbreaks mostly rely on base64/leetspeak obfuscation, iteratively optimized adversarial strings, or specific prompt templates; this paper shows that a much simpler past-tense rewrite suffices, indicating that alignment techniques such as SFT, RLHF, and adversarial training may be brittle and do not always generalize as intended.
   • Core observation: the authors believe refusal guardrails tend to treat past-tense historical questions as more benign than hypothetical future questions; the markedly weaker effect of future-tense rewrites supports this conjecture.
   • Paper contributions: systematically evaluate the past-tense attack on 8 models; compare with future tense; verify that adding past-tense refusal samples to fine-tuning data can defend; evaluate the o1 reasoning models; and discuss the blind spots of alignment methods from a generalization perspective.
   • Threat model:
     • Goal: find a prompt P such that the target LLM's output is judged by the semantic judge JUDGE as a valid jailbreak for the harmful request R.
     • Knowledge and capability: the attacker uses GPT-3.5 Turbo to automatically rewrite requests, with 20 rewrite attempts per request and temperature 1; the paper states that the attack is universal and transferable and does not rely on iterative optimization or additional prompting techniques.
     • Victim systems: Llama-3 8B, Claude-3.5 Sonnet, GPT-3.5 Turbo, Gemma-2 9B, Phi-3-Mini, GPT-4o-mini, GPT-4o, R2D2, and o1-mini, o1-preview.
     • Success criterion: success if at least one of the multiple attempts yields an unsafe response, as judged by GPT-4, Llama-3 70B, or the rule-based judge.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: alignment generalization research, adversarial attacks targeting refusal mechanisms, and the rewrite-based and multi-turn attacks most similar to this paper.

   Generalization of alignment

   • Dang et al. (2024) and Li et al. (2024b)—observed respectively that RLHF and refusal training generalize from English to other languages; Wendler et al. (2024) argue that English-dominant pretraining maps other languages internally to English. The paper notes this contrasts with past-tense rewrites, and hypothesizes the reason is that different tenses have different internal representations.
   • Berglund et al. (2024)—propose the reversal curse: training on "A is B" does not automatically generalize to "B is A", showing that generalization mechanisms humans take for granted do not necessarily appear in LLMs.

   Adversarial attacks targeting refusal mechanisms

   • Wei et al. (2023a)—divide manual jailbreaks into competing objectives and mismatched generalization; the paper explicitly classifies this paper's attack as the latter.
   • Zou et al. (2023)—use the gradient method GCG to generate universal transferable adversarial suffixes; Lapid et al. (2023) and Liu et al. (2023) use genetic algorithms, Zhu et al. (2023) makes low-perplexity modifications, and Andriushchenko et al. (2024), Sitawarin et al. (2024) and Hayase et al. (2024) use random search. The paper states that this paper's method is conceptually much simpler: direct rewriting, with no iterative optimization.
   • Wei et al. (2023b), Anil et al. (2024)—enhance jailbreaks with in-context learning prompts containing examples of harmful responses.

   Closest work

   • Perez et al. (2022), Chao et al. (2023)—use an auxiliary LLM to generate multiple rewrites of harmful requests, sharing the same origin as this paper's approach of rewriting with GPT-3.5 Turbo.
   • Russinovich et al. (2024)'s Crescendo multi-turn attack—first asks multiple benign questions then poses the target question, and in its example the final step happens to be a past-tense question; the paper notes that this step alone is often sufficient and multi-turn dialogue is not necessary.

   Baselines and benchmarks: The comparison baselines are the transfer attack of request-specific GCG suffixes optimized by Chao et al. (2024) on Vicuna, and the comparison with Andriushchenko et al. (2024) on the same 50 AdvBench behaviors; the evaluation benchmark is JBB-Behaviors (100 harmful behaviors, drawn from AdvBench, the Trojan Detection Competition and its own samples). The authors position this paper as using the simplest tense rewrite to probe blind spots in alignment generalization, rather than pursuing a SOTA jailbreak method.

   Collapse this answer

3. 03How does the paper address this problem?

   GPT-3.5 Turbo is used to automatically rewrite harmful requests into the past tense, with 20 attempts per request, and a semantic judge determines whether the jailbreak succeeds.

   Overall approach: rewrite harmful requests from present tense into past-tense questions, use sampling diversity to make multiple attempts, and count the attack as successful if even one attempt yields an unsafe response. The method is not separately named; the paper calls it the past-tense reformulation attack.

   Problem formalization

   • The target model is defined as LLM : T* → T*, and the semantic judge as JUDGE : T* × T* → {NO, YES}.
   • The attacker's goal is: find P ∈ T^∗ s.t. JUDGE(LLM(P), R) = YES, i.e., find a prompt P such that the output is judged to be a valid jailbreak for the harmful request R.

   Rewrite module

   • GPT-3.5 Turbo is used with the prompt in Table 2 to rewrite arbitrary requests into past-tense questions; the prompt contains a few examples. The authors consider this type of rewrite conceptually simple and see no need to use a stronger model.
   • Both the target model and the rewrite model have temperature set to 1, and sampling diversity is used to make 20 rewrite attempts per request.
   • The paper notes that the attack could be further strengthened by combining it with known techniques such as refusal suppression and competing objectives, but deliberately does not use them here, because the goal is to demonstrate a generalization gap rather than to build a SOTA attack.

   Evaluation setup

   • Data: the 100 harmful behaviors in JBB-Behaviors, covering 10 harm categories defined by the OpenAI usage policy; samples come from AdvBench, the Trojan Detection Competition, and samples unique to JBB.
   • Judges: the primary judge is GPT-4 (following the method of Chao et al. (2023), scoring 1–10); to avoid overfitting to a single judge, a Llama-3 70B judge (the prompt from Chao et al. (2024)) and the rule-based judge of Zou et al. (2023) are also used; judge prompts are in Appendix A.
   • System prompts: Llama-3 8B uses the Llama-2 refusal-augmented system prompt, and the other models use their respective default system prompts (listed in Appendix A).

   Control and extended experiment design

   • Future-tense control: the prompt in Table 10 is used to have GPT-3.5 Turbo rewrite requests into the future tense, with all other settings the same.
   • Fine-tuning defense: fine-tuning is done on gpt-3.5-turbo-0125 via the OpenAI fine-tuning service, using 394 past-tense rewrite paired refusal responses from 50 randomly selected JBB-Behaviors, with the other 50 as the test set; OpenHermes-2.5 standard dialogue is mixed in at ratios of 2%/98%, 5%/95%, 10%/90%, and 30%/70% (refusal data/standard data).
   • Over-refusal measurement: the refusal rate is measured on the 100 borderline benign behaviors in JBB-Behaviors, using a Llama-3 8B judge (Table 13 prompt) to detect refusals.
   • o1 evaluation: the same past-tense/future-tense attacks are run on o1-mini and o1-preview, and interception by their input filters is observed.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The past-tense attack raises ASR on 8 models to mostly 53%–98% (GPT-4 judge); future tense is weaker, and fine-tuning can defend but pushes up over-refusal.

   Experimental Setup

   • Models tested: Llama-3 8B Instruct, Claude-3.5 Sonnet, GPT-3.5 Turbo, Gemma-2 9B Instruct, Phi-3-Mini Instruct, GPT-4o-mini, GPT-4o, R2D2; also tested o1-mini and o1-preview; the fine-tuning experiment subject was gpt-3.5-turbo-0125.
   • Dataset: 100 harmful behaviors from JBB-Behaviors (10 harm categories); for fine-tuning, 50 were used for training and 50 for testing; over-refusal used 100 borderline benign behaviors.
   • Baselines: the transfer attack of request-specific GCG suffixes optimized by Chao et al. (2024) on Vicuna; compared with Andriushchenko et al. (2024) on the same 50 AdvBench behaviors.
   • Metrics: ASR (success if at least one of 20 rewrite attempts yields an unsafe response); the fine-tuning experiment also measured overrefusal rate.
   • Judging methods: three judges—GPT-4 (1–10 scoring), Llama-3 70B, and rule heuristics; refusal detection used Llama-3 8B.
   • Sampling settings: both target and rewrite models had temperature 1, with 20 attempts per request; the paper did not report the number of repeated experiments.

   Main Results

   According to Table 1 (100 JBB-Behaviors, 20 past-tense rewrite attempts, ASR as direct request → past tense):

   Swipe horizontally to read the table

   Model	GPT-4 judge	Llama-3 70B judge	Rule judge
   Llama-3 8B	0% → 27%	0% → 9%	7% → 32%
   Claude-3.5 Sonnet	0% → 53%	0% → 25%	8% → 61%
   GPT-3.5 Turbo	0% → 74%	0% → 47%	5% → 73%
   Gemma-2 9B	0% → 74%	0% → 51%	3% → 68%
   Phi-3-Mini	6% → 82%	5% → 41%	13% → 70%
   GPT-4o-mini	1% → 83%	1% → 66%	34% → 80%
   GPT-4o	1% → 88%	1% → 65%	13% → 73%
   R2D2	23% → 98%	21% → 56%	34% → 79%

   • The authors state that the ASR of GPT-4o-mini and GPT-4o are close, suggesting they may use the same alignment method; GPT-3.5 Turbo (74%) is slightly more robust than GPT-4o (88%).
   • The authors state that R2D2's direct-request ASR was already 23%, rose to 98% with past tense, and adversarial training did not block this attack.
   • Figure 2 shows ASR rising with the number of attempts: GPT-4o reaches 57% with a single attempt, and most models saturate after 10 attempts; the authors use this to justify choosing 20 attempts.

   Comparison with Existing Attacks

   • Under the same evaluation setup, the GCG transfer suffix optimized on Vicuna achieves 47% ASR on GPT-3.5 Turbo and only 1% on GPT-4o (Llama-3 70B judge); the authors state this shows frontier models have patched known attacks but remain vulnerable to new ones.
   • On the same 50 AdvBench behaviors as Andriushchenko et al. (2024), this paper's attack reaches 100% ASR on GPT-4o with 100 restarts, which the authors state is competitive with SOTA.

   Future-Tense Comparison and Category Analysis

   • Future-tense rewrites (Table 3, GPT-4 judge): GPT-4o is 1% → 61%, Gemma-2 9B is 0% → 59%, R2D2 is 23% → 94%, Claude-3.5 Sonnet is only 0% → 5%; the authors state future tense is overall weaker than past tense but still higher than direct requests.
   • The authors propose two hypotheses to explain the tense difference: harmful requests in fine-tuning data appear more often as future-tense or hypothetical events; or models interpret future-oriented requests as more dangerous and past tense as more benign.
   • Figure 3 breaks down by 10 harm categories (GPT-4 judge): the authors state most models have ASR near full marks on malware/hacking, economic harm, fraud/deception, and government decision-making, and lower on harassment, disinformation, and adult content; the authors suggest this may be because the latter categories contain more conspicuous sensitive words, making refusals more likely.

   Fine-Tuning Defense Experiment

   According to Table 4 (gpt-3.5-turbo-0125 fine-tuning, past-tense ASR and over-refusal rate):

   Model	Over-refusal rate	GPT-4 judge ASR
   GPT-3.5 Turbo (not fine-tuned)	3%	0% → 74%
   FT 2%/98%	6%	2% → 24%
   FT 5%/95%	22%	0% → 0%
   FT 10%/90%	32%	0% → 2%
   FT 30%/70%	61%	0% → 0%

   • The authors state that adding past-tense refusal samples to fine-tuning data can directly reduce ASR to 0%, but the over-refusal rate rises with the proportion of refusal data.
   • Reference point: Llama-3 8B has an over-refusal rate of 19% and ASR of 27% (GPT-4 judge); the authors state that FT 2%/98% (6% over-refusal, 24% ASR) improves the Pareto frontier between correct refusals and incorrect refusals.

   o1 Reasoning Models

   • According to Table 5 (GPT-4 judge): o1-mini past-tense ASR is 3% → 84%, future tense 3% → 45%; o1-preview past tense 2% → 78%, future tense 2% → 56%.
   • The authors note that o1's jailbreak outputs often do not meet the Model Spec's definition of jailbreak: many are "dual-use" content, such as giving only general writing advice rather than a finished piece, and not always giving detailed recipes, making them less useful to attackers.
   • o1 is equipped with an input filter: on average it blocks 78% of present-progressive requests and 27% of past-tense requests; the authors state that past-tense rewrites also help bypass the filter, but they had no access to the filter and did not optimize against it.
   • The authors observe that in successful jailbreaks, the reasoning summary first acknowledges the request may be harmful but continues answering because it is a "historical reference"; they also suggest that the natural break between chain of thought and final answer acts like an implicit output filter.

   Other Ablations and Analyses

   • Differences between judges: ASR under the Llama-3 70B judge is generally lower than under the GPT-4 judge (e.g., GPT-4o 65% vs 88%, Table 1).
   • Under the rule judge, GPT-4o-mini's direct-request ASR already reaches 34%, higher than other models (Table 1).
   • Failure cases: the authors observe that attacks are harder to succeed when requests are highly specific (e.g., writing a poem praising a particular event), and easier when the required knowledge is more general.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors hypothesize that tense representation differences cause the generalization failure and call for research on alignment generalization mechanisms; the experiments cover 10 models and a single rewrite model.

   Limitations and future directions noted by the authors

   • The authors note that many of o1's jailbreak generations do not meet the Model Spec's definition of a jailbreak and are more generic in information, and that the usefulness of generated answers should be evaluated in a more fine-grained way than in existing literature (Section 5, Discussion).
   • The authors hypothesize that the generalization failure stems from different languages mapping to similar representations while different tenses require different representations, but this is only a hypothesis (hypothesize) and no representation-level verification was done (Section 7).
   • The authors believe the generalization mechanisms of current alignment methods are understudied and require further research (Section 7).
   • The authors propose that the past-tense blind spot raises an open question: what other blind spots exist in current alignment techniques, and why do they persist (Section 1).
   • The authors state that with more data in the fine-tuning defense, the trade-off between refusal and over-refusal may improve further (Section 4).
   • The authors mention alternative defense directions that do not rely on SFT/RLHF: output detectors and representation-based methods (harmfulness probing, representation rerouting), but these are not implemented in this paper (Section 7).

   Scope of experiments

   • There are 10 models tested: 8 aligned models (including the adversarially trained R2D2) plus o1-mini and o1-preview; the fine-tuning defense is done only on gpt-3.5-turbo-0125.
   • Only GPT-3.5 Turbo is used as the rewrite model; each request has a fixed 20 rewrite attempts, with temperature 1.
   • The evaluation data are the 100 harmful behaviors in JBB-Behaviors (English, 10 harm categories); over-refusal uses 100 borderline benign behaviors.
   • There are three judges: GPT-4, Llama-3 70B, and a rule-based heuristic; the paper does not report agreement between judges and human judgments, nor the number of repeated experiments.
   • The authors explicitly do not use enhancement techniques such as refusal suppression and competing objectives, nor do they optimize against o1's input filter.

   Collapse this answer

6. 06Summarize the main content of the paper

   Past-tense rewrites can bypass the refusal training of multiple frontier LLMs; future tense is weaker; fine-tuning with past-tense refusal samples can defend.

   An ICLR 2025 conference paper studying the generalization gap of refusal training under tense rewrites, belonging to jailbreak attacks and alignment generalization analysis.

   • Problem: The authors state that alignment techniques such as SFT, RLHF and adversarial training generalize to many unseen rewrites, but merely rewriting harmful requests into past-tense questions is often enough to jailbreak multiple frontier models.
   • Method: Use GPT-3.5 Turbo to automatically rewrite requests into past tense, with 20 attempts per request (temperature 1), success if at least one unsafe response occurs; judged on 100 harmful behaviors from JBB-Behaviors by three judges: GPT-4, Llama-3 70B and rules.
   • Main results: Under the GPT-4 judge, GPT-4o's ASR rises from 1% to 88%, R2D2 from 23% to 98%, Claude-3.5 Sonnet from 0% to 53%; o1-mini and o1-preview reach past-tense ASR of 84% and 78% respectively, but the authors note their jailbreak outputs are more generic.
   • Comparison: Future-tense rewrites are overall weaker (GPT-4o 61%, Claude-3.5 Sonnet only 5%); the authors infer from this that guardrails treat historical questions as more benign than hypothetical future questions.
   • Defense: Mixing past-tense refusal samples into gpt-3.5-turbo-0125 fine-tuning can reduce ASR to 0% (FT 5%/95%), but the over-refusal rate rises to 22%, which needs to be controlled with standard dialogue data.
   • Authors' conclusion: Alignment techniques generalize across languages but not across tenses; the authors hypothesize the reason is that different tenses have different internal representations, and suggest such simple jailbreaks can serve as tools to probe blind spots in alignment generalization.

   Collapse this answer

Related papers