Researchers propose a jailbreak attack method that extracts the safety classifier of an aligned LLM

Alignment embeds whether to refuse into the LLM, but white-box jailbreaks are costly and rely on heuristics; the authors propose extracting an approximation of this safety classifier and attacking it.

• Setting: The authors state that alignment is used to enforce criteria such as safety, but it fails under adversarial inputs, producing jailbreaks: unsafe inputs are accepted as safe. They state that white-box attacks have high runtime and memory usage, and use heuristics (e.g., maximizing the likelihood of a particular output sentence, or making fixed changes to the input), narrowing the search space of adversarial inputs.
• Core assumption: The authors assume that alignment embeds a safety classifier in the LLM that decides whether an input is safe (compliant) or unsafe (refusal). They assume that the final intermediate state before output is sufficient to determine whether the output is a refusal.
• Approach in this paper: They construct candidate classifiers from a subset of the LLM's structure, check whether they track the model's decisions in benign and adversarial settings, select a surrogate, then attack the surrogate and transfer the adversarial inputs back to the LLM to evaluate the robustness of alignment to adversarial inputs.
• Threat model:
  • Knowledge: The paper specifies white-box, i.e., requires model weights. Section V-B states that the threat model is limited to white-box.
  • Goal: To extract an approximation of the safety classifier of the target aligned model. The jailbreak itself is adding perturbations to unsafe inputs so that the model complies. The authors state that once extraction succeeds, subsequent white-box jailbreaks can avoid the computational cost of attacking directly.
  • Capabilities and actor: They select a subset of the model, train a classification head, and generate adversarial suffixes against the surrogate. The authors state that the actor can be a red-teamer auditing the model.
  • Victim system and judgment: The victim system is an aligned chat LLM. Classification looks at whether the output is a refusal or compliant, not whether the output contains unsafe information. Success is measured by the misclassification proportion ASR, and the transferability rate at which adversarial inputs made on one side are misclassified by the other.

The introduction, background, and Section VI divide related work into four groups: alignment judgment, jailbreak attacks, representations and components, and pruning.

• Alignment and output judgment: SFT fine-tunes with human responses; RLHF (Christiano et al., 2017; Bai et al., 2022) additionally trains a reward network; DPO (Rafailov et al., 2023) uses the LLM itself as a reward model. There are taxonomies for unsafe behaviors, and the paper cites Llama Guard (Inan et al., 2023). Early work used refusal keywords to judge outputs (Zou et al., 2023); there are also LLM judges with varying granularity (Inan et al., 2023; Mazeika et al., HarmBench, 2024). The paper notes that judges can also be attacked, potentially classifying unsafe outputs as safe (Mangaokar et al., 2024; Raina et al., 2024).

• Jailbreak attacks: On the white-box side, GCG (Zou et al., 2023) uses greedy coordinate gradient to handle the lack of a direct mapping from embeddings to tokens; subsequent work improves the algorithm, changes the objective, or exploits logits, special tokens, and the generation process. The paper states that GCG-type attacks achieve an average ASR above 50% on most open-source models (paraphrasing Mazeika et al., 2024). On the black-box side, the paper describes character art transformations, low-resource languages, existing templates, and search algorithms, and states that ArtPrompt can achieve an ASR as high as 80% on GPT-3.5. These are the paper's paraphrases of existing attacks, not experiments in this paper.

• Representations, probes, and components: The paper states that refusal and acceptance must be linearly separable in some representations because the output explicitly depends linearly on the last embedding (Lin et al., 2024); separation has been reported in decoder layers and attention heads. Linear probes have been used to classify these representations, but the paper notes they lack gradients from embeddings to tokens and cannot directly produce token-level adversarial examples. Existing uses include ablation, intervention on embeddings, and optimization over inputs; the paper states that perturbations from intervention do not necessarily correspond to feasible input perturbations.

• Pruning: Depth, width, and length pruning can be combined. The paper mentions that safety alignment can be isolated by low-rank structure (Wei et al., 2024), and pruning can also improve refusal robustness (Hasan et al., 2024). The paper notes that these techniques typically modify selected parts of the model.

• Baseline method: Direct use of GCG's nanogcg implementation on the full LLM. Benchmarks/datasets: an enhanced version of AdvBench, and a random subset of OR-Bench.

The authors position this paper as: viewing alignment as a classifier inside the model and extracting it, enabling jailbreaks to produce token-level adversarial inputs. Compared to probes that only train on representations and cannot backpropagate to tokens, and pruning that modifies model components, the authors state they choose linear probing to reduce training and to lose less information on the decision boundary.

Overall, the authors reduce jailbreaking to misclassification of the embedding of a safety classifier: they use a subset of the model plus a classification head to obtain a surrogate classifier, attack it with a misclassification objective, and transfer the adversarial inputs back to the LLM.

Problem Setting

• Task: fθ is an LLM with context window N and vocabulary V. A rule R maps outputs to classes. The safety task is binary classification: unsafe prompts are the positive class (corresponding to refusal), safe prompts are the negative class (corresponding to compliance). Jailbreaking is a false negative, and false positives are over-refusal.
• R in this paper: The authors use a classifier to determine whether the output is a refusal or compliance, focusing on the type of response rather than whether the output contains unsafe information.
• Extraction target: A structure fθ* (where θ* is a subset of θ) plus a classification head C such that for any input x, R(f_θ(x))=C(f_θ^∗(x)). This equation indicates that the candidate and the LLM agree on this classification task.
• Separation observation: The authors use the silhouette score to see whether the two classes of embeddings are separated. The score ranges from -1 to 1; the criteria they cite are that below 0 indicates overlap, above 0.25 or 0.5 indicates weak or moderate separation. In Figure 2, the x-axis is normalized decoder position, and the y-axis is Silhouette Score (scale 0.0 to 0.6). The authors state that the score decreases after reaching a maximum, with the same trend across four models and different magnitudes across two datasets, and they interpret the decrease as later layers adding information unrelated to safety.

Extracting a Candidate from the Structure

• Three steps: select a structure; collect structure features and R(fθ(x)); train a classification head. The result is called a candidate classifier. The authors state that they keep both correct and incorrect labels because the goal is to extract a classifier, not to improve it.
• Structure: The authors assume the classifier is at the decoder level and that relevant information is at the end of the sequence, so the classification head only looks at the last vector of the structure output, with input dimension d. The structure is denoted fi,i+δ, where δ is the candidate size and Dδ is the proportion of the LLM's decoders. In this paper, i is fixed to 1, meaning consecutive decoders starting from the first layer. Isolating intermediate decoders is left for future work.
• Classification head: To avoid overfitting with a complex head, they use linear probing: the output is sigmoid(Ah+b), then thresholded by T into two classes. T is chosen to maximize F1 on the training set.
• Training: 5-fold cross-validation. Table I gives learning rate 0.001, batch size 32, epochs 500, Adam, patience 15, and 5 folds. During generation, temperature is 0, and the token with the highest logit is taken at each step. R uses distilroberta-base-rejection-v1.

Data and Attack Objective

• Data completion: When safe prompts are missing, an LLM is used to rewrite unsafe prompts into safe versions with the same structure, and two properties are manually checked: they do not request unsafe information and the structure is the same as the original prompt. The paper has example rewrites, not repeated here. The original GCG also requires a target output string; when missing, an LLM generates possible compliant outputs. The LLM responsible for these two steps is not named in the paper.
• Attack pipeline: Use GCG to generate adversarial examples on the candidate, then see whether they cause the LLM to misclassify. The authors state this is similar to transfer from a different source model, but the source is a subset of the target LLM.
• Objective rewrite: The original GCG maximizes the log probability of a target token sequence. The authors argue that the space of compliant outputs is infinite, and tying the search to a single sentence limits effectiveness. Once the candidate provides safety labels, the objective is changed to maximize L(C(f_θ^∗(x₁… x_T+A)), y) over the adversarial suffix, where y is the LLM's predicted label and L is, for example, binary cross-entropy. The authors state that if the candidate satisfies the consistency equation above, increasing the likelihood of compliant sentences means getting closer to being classified as safe, so under a perfect candidate this objective subsumes the original GCG objective.
• Implementation choices: nanogcg, num_steps=250, topk=512, search_width=512, attack length 20; on the candidate, PyTorch binary cross-entropy is used. The main experiments push unsafe inputs toward compliance. The authors do not use BEAST or AutoDAN as primary attacks, reasoning that the former is not gradient-based and the latter's optimization objective is hard to map to misclassification; they also do not use AmpleGCG, reasoning that it adds complexity and does not align with the goal of removing heuristics.

实验设置

• 模型：主实验四个开源对齐 chat 模型，Transformers 4.46.2：Llama-2-7b-chat、Qwen2.5-7B-Instruct、gemma-7b-it、granite-3.1-8b-instruct。图表简称依次为 Llama 2、Qwen 2.5、Gemma 1、Granite。附录 C 另报 Llama-3.1-8B-Instruct、Mistral-7B-Instruct-v0.3、Zephyr_RMU 的良性 F1，未报告对它们的攻击迁移数字。
• 数据：增强 AdvBench 为 520 条原不安全指令加 520 条安全指令。OR-Bench 随机 1000 条，不安全与安全各 500；作者称出于计算原因抽样。
• 标签与指标：temperature 为 0。拒绝或合规由 distilroberta-base-rejection-v1 判定。candidate 用 F1。ASR 是被攻击改过的输入里被模型误分类的比例。transferability rate 是在一方上做出的对抗输入被另一方误分类的比例。
• 协议：分类头 5 折；Figure 4 的 Test F1 为 5 次试验的中位数。攻击为 nanogcg 版 GCG，num_steps=250，topk=512，search_width=512，长度 20。对抗评测用各折里分类头没见过的提示，阈值取训练 F1 最大者。
• 算力：NVIDIA A100，40 GB 显存。作者称对模型的攻击约 2000 GPU 小时，对 candidate 约 500 GPU 小时。效率为 100 个样本的平均。分类头训练作者称不足 1 分钟，且对全部样本只做一次。
• 对照：直接对完整 LLM 跑同一 GCG。作者称原目标每步都要生成输出序列并与目标序列比较。

主结果

Swipe horizontally to read the table

模型（表中名）	AdvBench 直接 ASR	OR-Bench 直接 ASR	50% 结构迁回 LLM
Gemma 1	0.62	0.67	未报告
Granite	0.91	0.88	未报告
Llama 2	0.22	0.12	70%（未写数据集）
Qwen 2.5	0.94	0.88	未报告

前两列据 Table III。70% 出自正文与 Takeaway 6，指攻击 Llama 2 的 50% 结构后迁回该 LLM；该句未写是 AdvBench 还是 OR-Bench。摘要和引言里的直接攻击 22% 同样未在该句写数据集。

• 直接攻击：作者写 Llama 2 与 Qwen 2.5 的 ASR 相差约 70%，并归因于 Llama 2 对齐强、偏向拒绝（见 Section A）。除 Gemma 1 外，OR-Bench 上的直接 ASR 低于 AdvBench；作者称这可能与 OR-Bench 强调 over-refusal 有关。Gemma 1 相反：AdvBench 0.62，OR-Bench 0.67。
• surrogate：作者称 Figure 9 不随 candidate size 单调上升。其中三个模型在 normalized candidate size 接近 60% 时峰值高于 70%，论文未点名是哪三个。后文又称所有模型只用一半结构时迁移率高于 70%。Llama 2 的具体数字是 50% 结构、ASR 70%，作者称高于直接攻击基线。
• 目标函数：作者称差别来自对抗目标。文本输出需要用特定目标句判断越狱；改成误分类后搜索空间更大。δ 太小抓不住攻击所需信息，δ 太大则会掺入与安全分类正交的信息。

良性一致与跨数据集

• 同分布：作者称 Figure 4 中性能随 candidate size 上升或持平，且四个模型在 normalized candidate size 低至 20% 时 F1 高于 80%。OR-Bench 上 candidate 的 F1 更低，作者写所有模型 losing 5%，未说明是百分点还是相对降幅。Table II 中 LLM 自身 F1：AdvBench 上 Gemma 1 与 Qwen 2.5 为 0.97，Granite 0.94，Llama 2 0.93；OR-Bench 上依次 0.95、0.89、0.88、0.83。
• 跨数据集：Figure 5 是在未参与训练的数据集上的中位 F1。作者称最大 candidate size 下换分布后下降不大，但收敛更慢。Gemma 1 在 Figure 4a 中，normalized size 高于 20% 时 F1 为 0.9；Figure 5a（AdvBench 训练、OR-Bench 测试）要到 50%。作者认为 candidate 会过拟合数据集格式，另一数据集给出的是 δ 的下界。

LLM 到 candidate 的对抗迁移

• 样本量：先攻击 LLM，只保留被 LLM 误分类的对抗输入。Figure 8 标注：AdvBench 上 Gemma 1 n=320，Granite n=473，Llama 2 n=113，Qwen 2.5 n=487；OR-Bench 上依次 n=334、443、59、441。
• 作者对 Figure 8 的表述：迁移率至少用到一半模型才收敛。正文写 Llama 2 只有超过一半模型时迁移率才高于 90%；Takeaway 4 写使用超过 50% 时超过 95%。两处阈值不同，正文未给逐点读数。
• 数据集差异：作者称 OR-Bench 上迁移率收敛到 80% 以上，AdvBench 为 60% 到 80%，Llama 2 是唯一例外。OR-Bench 上使用不到 40% LLM 的多数 candidate 不稳定。作者把 OR-Bench 上更高的收敛值归因于被判为不安全的样本更多；并称 Llama 2 不受此趋势影响，因为它把安全输入误判为不安全的比例最高。

效率

• 引言中的 Llama 2：50% 结构对照直接攻击，作者写 350s/sample 对 850s/sample，14 GB 对 31 GB，并写 less than half。摘要则写 half the memory footprint and runtime，未给出这四个数。
• Figure 10：图注写虚线为直接攻击 LLM 的平均基线 840s 与 31 GB；曲线是随 candidate size 制作一个对抗样本的 runtime 与 VRAM，对 100 个样本平均。图注未写模型名。作者称二者都未达到 31 GB 与 840s，并把基线更慢归因于每步生成输出序列。作者称攻击 50% 模型时 20 GB 显存即够；这句是作者的推论，不是图上标注的读数。
• 参数规模：作者称所评 LLM 参数量小于 9 billion，使用 50% 即够做这一攻击。

其他消融与分析

• 附录 B 改为攻击安全输入、诱使拒绝。Table IV 的 ASR：Gemma 1 为 AdvBench 0.4、OR-Bench 0.47；Granite 0.68、0.7；Llama 2 均为 0.99；Qwen 2.5 为 0.88、0.85。作者称 Llama 2 在不安全输入上 ASR 最低，在安全输入上最高。
• 附录 B：作者称迁移趋势与主实验相似，多数模型有峰值，Llama 2 例外；OR-Bench 上 candidate 的 ASR 与迁回 LLM 的比例一致。拒绝目标句见附录，此处不复述。
• 附录 C、Table V，模型自身 F1：Llama 3 为 AdvBench 0.84、OR-Bench 0.63；Mistral 0.74、0.77；Zephyr RMU 0.7、0.62。作者称三者在 OR-Bench 上 F1 都低于 0.8。
• 附录 C：作者称 Figure 14a 在某一 candidate size 后收敛，但 F1 低于主实验模型；Figure 14b 在模型中部下降。Figure 15 为跨数据集曲线，正文未给逐点数字。
• 作者对附录 C 的解释：所引 Mistral 模型卡称该演示没有 moderation mechanisms；Llama 3 调整了拒绝语气，使检测分类更难；Zephyr RMU 经 unlearning 后仍可能对不安全输入合规并给出安全输出，作者认为 unlearning 可能去掉 safety classifier 的一部分。
• Figure 2 的正文未给出各点 silhouette 数值。作者称 OR-Bench 的 silhouette 一致低于 AdvBench。

Limitations and future directions noted by the authors

• Intermediate-layer structure: This paper only studies f1,1+δ starting from the first layer. The authors write that i>1 requires new methods so that the candidate does not use information from earlier decoder layers, and they leave this exploration for future work (Section V-C).
• Temperature: The experiments set temperature to 0 to ensure reproducibility and reduce computation. The authors write that temperature and its effect on refusal and compliance probabilities are left for future work (Section V-C).
• Black-box: The authors write that this paper limits the threat model to white-box, because extracting a surrogate requires the weights; and they state that studying whether the safety classifier of a source LLM can transfer to another target LLM is a direction (Section V-B).
• Other alignment failures: Malicious code, hallucination, dishonesty, bias, and unfairness cannot be directly formulated as classification problems. The authors state that one could construct controlled environments, use prompts in a specific format to make the LLM act as a classifier, and then apply this approach (Section V-A).
• Weak alignment and unlearning: Appendix C writes that this method is not suitable for models that are weakly aligned or strengthened in ways different from traditional alignment. The authors consider models strengthened via unlearning to be orthogonal to the scope of this paper, because unlearning may remove part of the safety classifier (Appendix C).

Experimental coverage

• The tested models for the main results are Llama-2-7b-chat, Qwen2.5-7B-Instruct, gemma-7b-it, and granite-3.1-8b-instruct, 4 models in total. Appendix C additionally reports benign F1 and cross-dataset curves for Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and Zephyr_RMU; the paper does not report attack ASR or transferability for these 3 models.
• The datasets are augmented AdvBench (520 unsafe plus 520 safe) and a random 1000 from OR-Bench (500 each). The labeler is distilroberta-base-rejection-v1, and the generation temperature is 0. The paper does not report agreement between this labeler and human judgment, nor does it name the LLM used for data augmentation.
• The white-box attack algorithm is GCG (nanogcg), with hyperparameters num_steps=250, topk=512, search_width=512, and length 20. The authors explain in Section IV-A that BEAST is not gradient-based unconditionally and AutoDAN's objective is difficult to map to misclassification, so they are not used as the main attack; AmpleGCG is not used because it adds complexity and does not align with the goal of removing heuristics.
• The structure is a contiguous prefix starting from the first decoder. Benign F1 uses 5 folds, and Figure 4 and Figure 5 report the median over 5 trials. Figure 8 gives the n used to compute transferability; Figure 10 averages over 100 samples. The paper does not state the sample size per point in the caption of Figure 9.
• Appendix B reports the ASR of attacking safe inputs into refusals (Table IV). Section V-B discusses adversarial training of the safety classifier, enlarging the classifier, and using it at inference time to discard unsafe inputs early; the paper does not report experimental results for these defenses.

The authors propose a white-box jailbreak method: extract an approximation of the safety classifier inside an aligned LLM, attack that approximation, and transfer back to the original model.

• Problem: Alignment should cause unsafe inputs to be rejected, but jailbreaks can cause such inputs to be accepted. The authors argue that direct white-box attacks are expensive, and heuristics such as target sentences limit the search.
• Method: Take consecutive decoders from the first layer, add a linear classification head, and fit whether the model output is a refusal or compliance. Then replace GCG's target sentence likelihood with a misclassification objective and attack the selected surrogate.
• Benign: The authors state that on the four main models, the candidate can achieve F1 above 80% with as little as 20% of the structure. In Table II, the F1 of these LLMs themselves on AdvBench is at least 0.93 for Llama 2 and at most 0.97 for Gemma 1 and Qwen 2.5.
• Attack: In Table III, the direct GCG AdvBench ASR is at least 0.22 for Llama 2 and at most 0.94 for Qwen 2.5; OR-Bench is at least 0.12 and at most 0.88. The authors state that Llama 2 with 50% structure has a transfer ASR of 70%, without specifying the dataset in that sentence; the direct attack in the abstract is 22%, also without specifying the dataset in that sentence.
• Overhead: The introduction writes the above Llama 2 setting as 350s/sample and 14 GB, compared to 850s/sample and 31 GB; the abstract writes half the memory and runtime. The Figure 10 caption gives the average baseline as 840s and 31 GB, without naming the model.
• Authors' conclusion: The authors believe that such approximations are sufficient to inspect the safety of aligned deployments and provide a route for attacks on alignment; they also believe this can reduce memory and runtime when studying alignment robustness. Other alignment failures such as hallucination and unfairness, the authors believe, could also be examined with the same approach.