Skip to content
Original

Weak-to-Strong Jailbreaking on Large Language Models

Weak-to-Strong Jailbreaking on Large Language Models

arXiv· arXiv:2401.17256· ICML 2025· Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, William Yang WangUC Berkeley; UC Santa Barbara; Sea AI Lab; Carnegie Mellon University; UC San DiegoIndexed by this site · 16 days ago· Originally publishedSelected AttacksSelectedAttention43
#Jailbreaks and attacks

Translation in progress — untranslated text is omitted.

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

AI summaryThe researchers propose a weak-to-strong jailbreak attack against aligned large language models, using one safe small model and one unsafe small model to modify the decoding distribution of a large safe model, with the target model requiring only one forward pass. The paper's analysis finds that the distribution difference between the safe model and the jailbroken model is concentrated in the first few tokens, and existing safety alignment is relatively shallow. On five open-source models (13B to 70B) from Llama2, Vicuna, Baichuan2, and InternLM, the method raises the attack success rate to above 99% on AdvBench and MaliciousInstruct, outperforming prior methods such as GCG; the 1.3B Sheared-LLaMA can also attack Llama2-70B-Chat on AdvBench with a 74.0% success rate. The authors' proposed gradient-ascent defense reduces the attack success rate by 5% to 10%, but they note that stronger defenses remain challenging; the attack requires access to output logits and requires a shared vocabulary, and the experiments are mainly on open-source models.

Questions
Existing automated jailbreak methods are computationally expensive and difficult to use on 70B-scale large models; the authors observe that the distribution difference between safety-aligned models and jailbroken models is mainly concentrated in the initial tokens, and alignment may only be shallow.
Methods
Proposes weak-to-strong jailbreaking: at decoding time, the log probability difference of a pair of 7B small models (safe and unsafe) sharing a vocabulary, scaled by a factor α, rewrites the large model's per-token distribution; the target model needs only one forward pass.
Experiments and results
On 5 open-source models, Llama2-70B achieves AdvBench ASR 99.2% and MaliciousInstruct 100% (Table 2); a 1.3B pruned model attacks 70B with 74.0% ASR; the gradient ascent defense proposed by the authors reduces ASR against this attack by only 5%–10%.
Limitations and future work
The authors state that the method requires access to output logits, and applicability to closed-source models is unverified; the defense remains a preliminary attempt. Experiments cover only 5 open-source models (7B–70B) and two main benchmarks; the paper does not report per-model sample sizes for each benchmark.
Why it was selectedThe paper shows that using a small model to modify a large model's decoding distribution can jailbreak it, with a single forward pass achieving over 99% success rate on multiple open-source models, revealing that existing safety alignment mainly acts on the initial tokens.

The paper shows that using a small model to modify a large model's decoding distribution can jailbreak it, with a single forward pass achieving over 99% success rate on multiple open-source models, revealing that existing safety alignment mainly acts on the initial tokens.

Related papers

Both use: SALAD-Bench
Show 3 more papers
Both use: MaliciousInstruct
Read originalICML 2025