Understanding the impact of safety unalignment on large language models

Compare whether, after unalignment via JT versus WO, the model merely refuses less, or is more able to assist in malicious activities while remaining useful.

• Scenario: The authors state that frontier models are generally safety-aligned before deployment, so as to refuse harmful requests while giving helpful and harmless responses. Unalignment makes the model complete requests it would otherwise refuse, potentially assisting malicious activities.
• Gap: The authors state that existing analyses are mostly limited to each method's individual refusal rate, lacking comparisons on adversarial attacks, cyberattacks, hallucination, reasoning models, and JT's helpfulness, and lacking a direct comparison between JT and WO.
• Observation: The authors state that a drop in refusal rate is a necessary but not sufficient condition for harmful capability. Therefore, one must see whether the unaligned model can perform malicious tasks, and whether hallucination and natural language capabilities are disrupted.
• Approach: Apply training-based JT and training-free WO simultaneously to six LLMs, comprising three instruction-tuned models and their reasoning counterparts, then perform supervised fine-tuning (SFT) on the WO models with benign data.
• Implementation conditions: The paper states in Section 4 that JT can be implemented at the fine-tuning interface via data poisoning; WO requires white-box weights to estimate the refusal vector and orthogonalize. The experiments in this paper use models with available weights from Table 1.

The introduction is grouped by method origin, with no separate Related Work section.

• Safety alignment: Bai et al. (2022), Dai et al. (2024), Ji et al. (2023), Tian et al. (2024), Wang et al. (2024) use multiple rounds of fine-tuning with malicious and benign data to make models refuse malicious requests and follow benign instructions. The paper calls DPO (Rafailov et al., 2023) and similar methods SOTA preference algorithms, and states that frontier models commonly undergo safety alignment before deployment (Dubey et al., 2024; Guo et al., 2025; Hurst et al., 2024; Yang et al., 2025).
• Training-based unalignment: Lermen et al. (2023) fine-tune with purely malicious data. Qi et al. (2024), Zhan et al. (2024) perform data poisoning; the paper states that even a small number of harmful samples can weaken guardrails. Bowen et al. (2025)'s jailbreak-tuning combines jailbreak instructions with harmful completions. The paper cites Betley et al. (2025), stating that JT weakens guardrails more than previous approaches, and reports: at a malicious/benign ratio of 2%, JT bypasses OpenAI moderation, fine-tunes gpt-4o, and reduces refusal capability on standard safety benchmarks by up to 95.8%. This number is not from this paper's experiments.
• Training-free orthogonalization: Arditi et al. (2024) estimate a refusal vector using harmful and harmless prompts, and orthogonalize weights to prohibit writing in that direction. The paper states this reduces refusal guardrails in non-reasoning models and does not substantially change natural language helpfulness.
• Baseline methods and benchmarks: The experimental baselines are the original aligned versions of each model; JT and WO serve as mutual comparisons, with no other attack algorithms used as comparisons. Evaluation uses StrongREJECT, AutoDAN-Turbo on HarmBench, CyberSecEval 3, TruthfulQA, TofuEval, and ARC-E, ARC-C, HellaSwag, PIQA, Winogrande, MMLU, IFEval. The paper states in Sections 3.3 and 3.4 that previous JT/WO TruthfulQA and helpfulness evaluations were limited to instruction-tuned models, and the two were not directly compared; JT's TruthfulQA evaluation was not compared with the base model, nor did it measure instruction following.

The authors position this paper as directly comparing JT and WO on harmful capability and helpfulness across the same set of instruction-tuned and reasoning models.

对同一批模型分别做 jailbreak-tuning（JT） 与 weight orthogonalization（WO），再只对 WO 模型做良性 SFT，比较有害协助与有用能力。

模型与比较轴

• Table 1 的六个模型：指令微调 Qwen3-4B（Qwen3-4B-Instruct-2507）、Llama-3.1-8B（Llama-3.1-8B-Instruct）、Qwen2.5-14B；推理对应 Qwen3-4B*（Qwen3-4B-Thinking-2507）、Llama-3.1-8B*（Deepseek-R1-Distill-Llama-8B）、Qwen2.5-14B*（Deepseek-R1-Distill-Qwen-14B）。* 表示推理模型。
• 比较拒答、对抗攻击、网络攻击协助、幻觉和通用 helpfulness。表中红色表示该变体最增加有害性或最降低 helpfulness。

JT

• 流程跟随 Bowen et al. (2025)。良性部分用 BookCorpus Completion（Pelrine et al., 2023）；越狱触发来自 PKU-SafeRLHF（Ji et al., 2025a）的恶意样本。论文未给出触发词原文。
• 最终 5,000 条，98% 良性、2% 恶意。微调 5 个 epoch，学习率 5e-4，adamw_torch，cosine annealing，weight decay 0.1，Q-LoRA（r=64，α=128，dropout=0.05）。

WO

• 免训练。在各层、指令后的各 token 位置，计算有害与无害指令的均值差，得到候选向量；选出移除后最能抑制恶意样本拒答的向量 r。
• 对写入残差流的权重正交化：W′ ← W − rr^⊤W。含义是阻止模型在残差流中写入拒答方向 r。
• 代码改自 Arditi et al. (2024)，为 Deepseek 与 Qwen3 增加拒答相关 token。有害指令随机取自 Huang et al. (2024)、Mazeika et al. (2023)、Zou et al. (2023)，无害指令取自 Alpaca。候选向量在恶意与良性验证指令上用 Llama Guard 2 打分。论文未写采样条数。

判定与评测协议

1. StrongREJECT：323 条恶意样本，temperature=0。评审是 StrongREJECT 专用的微调 Gemma-2B。
2. AutoDAN-Turbo：被评估模型作 attacker。目标是 Table 2 拒答率最高的两个对齐基座，Qwen3-4B 与 Llama-3.1-8B。scorer 为 Llama-3.1-8B。目标取 HarmBench 的 200 条训练样本。每目标最多 k=20 次，分数高于 8.5 算成功。ASR 是 200 个目标中的成功百分比。
3. CyberSecEval 3：1,000 条 MITRE ATT&CK 请求，看回复是否协助达成攻击目标。temperature=0.6，top_p=0.9，三个固定种子各生成一次后平均。附录 A.1 写明评审为 gpt-4o，10 类各 100 条。
4. 幻觉：TruthfulQA 的幻觉率是 1 减 MC1 与 MC2 准确率的平均。TofuEval 用 MeetingBank 的 50 篇文档、每篇 3 个主题，共 150 个摘要任务；temperature=0.7，评审 gpt-4o，每模型三次不同种子，幻觉率按全部 450 次摘要计算。
5. Helpfulness 用 Eleuther LM Evaluation Harness v0.4.9.2。ARC-E、ARC-C、MMLU、PIQA、Winogrande 报准确率，HellaSwag 报 normalized accuracy，IFEval 报 average accuracy。推理模型额外设置 think_end_token 为文中给出的结束标记。IFEval 与 TruthfulQA 使用 fewshot_as_multiturn 和 apply_chat_template；附录 A.2 称这两项 flag 会降低其余常识与推理任务的表现，故那些任务的最终结果未使用。

缓解用的 SFT

• 只对 WO 模型。数据为 OpenHermes，243,000 条。1 个 epoch，学习率 1e-5。其余参数沿用 Section 2 的训练配方；论文未逐项写明 SFT 是否使用 Q-LoRA。

Experimental Setup

• Models under test: The six in Table 1, each with original, JT, and WO; Section 3.5 additionally has WO-SFT. Short names correspond to original names as in the previous question.
• Benchmark sizes: StrongREJECT 323 items; HarmBench 200 training targets; CyberSecEval 3 1,000 items; TofuEval 150 tasks, 450 summaries. The paper does not report the number of samples for TruthfulQA and helpfulness items.
• Metrics: Refusal rate and % Decr.; ASR and % Inc. relative to the aligned model; hallucination rate and average percentage increase across two benchmarks; helpfulness and average percentage decrease.
• Evaluation: StrongREJECT uses fine-tuned Gemma-2B; AutoDAN-Turbo's scorer is Llama-3.1-8B with threshold 8.5; CyberSecEval 3 and TofuEval use gpt-4o. The paper does not report agreement between evaluators and humans.
• Replication: CyberSecEval averaged over three seeds; TofuEval three summaries per model. StrongREJECT uses greedy decoding. The paper does not report replication counts for AutoDAN-Turbo and helpfulness.
• Attack setup: The attacker is the unaligned model under evaluation; the victim models are the aligned Qwen3-4B and Llama-3.1-8B, not the unaligned models.

Main Results

Swipe horizontally to read the table

Model	Refusal Rate Orig/JT/WO	Attack Llama ASR JT/WO	Attack Qwen3-4B ASR JT/WO
Qwen3-4B	99.4 / 24.6 / 38.7	42.7 / 65.0	58.2 / 70.8
Llama-3.1-8B	98.4 / 24.9 / 9.9	62.4 / 64.9	59.5 / 65.2
Qwen2.5-14B	80.5 / 50.8 / 22.4	64.6 / 60.6	78.0 / 46.2
Qwen3-4B*	81.2 / 34.8 / 70.6	49.5 / 77.5	45.6 / 88.5
Llama-3.1-8B*	45.4 / 21.7 / 26.2	35.2 / 70.4	35.4 / 70.3
Qwen2.5-14B*	39.3 / 28.8 / 26.5	59.1 / 65.4	50.8 / 60.6

Refusal rates are from Table 2, ASR from Table 3. When a Llama row attacks Llama, the target is the same-named aligned base model, not another model family.

• The authors state that the refusal reduction is larger for each method on three models each. JT is larger on Qwen3-4B, Qwen3-4B*, Llama-3.1-8B*; WO is larger on Llama-3.1-8B, Qwen2.5-14B, Qwen2.5-14B*.
• The introduction states that relative to JT, WO succeeds against attacks by an average of 27.7% more; for reasoning models, an average of 40.2% more. Across the six reasoning model × target combinations, WO's ASR is always higher. The exception is Qwen2.5-14B: on both targets, JT is higher than WO.
• The authors state that six JT evaluations have ASR lower than the corresponding base model, four of which come from reasoning models. The % Inc. in Table 3 are −10.9, −0.3, −26.7, −39.2, −45.5, −43.4, corresponding to Qwen3-4B attacking Llama, Llama-3.1-8B attacking Qwen3-4B, and Qwen3-4B*, Llama-3.1-8B* on both targets. All twelve adversarial % Inc. for WO are positive.

Cyberattack Assistance

• On CyberSecEval 3 (Table 3), WO ASR is higher than JT for all six models. The authors state in the introduction that WO assists cyberattacks on average 6.1% more than JT.
• JT is lower than the base model on half the models: Llama-3.1-8B JT 97.7% (base 98.1%), Qwen3-4B* JT 60.1% (base 80.5%), Llama-3.1-8B* JT 88.1% (base 92.7%). WO is higher than the base model for all six models.
• The highest WO value in the table is Llama-3.1-8B WO at 98.6%; its base is already 98.1%. The largest relative increases are Qwen3-4B WO at 44.8% (from 67.7% to 98.0%) and Qwen2.5-14B* WO at 39.6% (from 59.0% to 82.4%).

Hallucination and Helpfulness

• Table 4: JT's average hallucination increase is higher than WO's across all six models. The authors state that JT raises hallucination rates on TruthfulQA and TofuEval by an average of 8.9 and 41.2 percentage points, while WO raises them by 3.6 and −1.7. The introduction states that WO has 39.5% less hallucination likelihood on average than JT. WO reduces TofuEval hallucinations on half the models and does not change Qwen2.5-14B (both 30.7). The largest Avg. % Incr. is Llama-3.1-8B* JT at 136.3%.
• Table 5: The authors state that across tasks and models on average, JT reduces helpfulness by 13.4%, WO by 2.2%, and call the latter a six-fold smaller decrease. The introduction states that WO preserves 11.2% more helpfulness. Counterexample: Qwen3-4B*'s avg. % decr. is WO 5.4, JT 2.8, so WO is larger. On IFEval, JT decreases more noticeably, e.g., Llama-3.1-8B from 79.9 to JT 30.9, WO 80.3.
• According to Table 4, all six WO TruthfulQA hallucination rates are higher than the corresponding base models. The authors state that both methods reduce refusal, but WO has a smaller impact on hallucination.

SFT Mitigation of WO

• Section 3.5 and Figures 1 and 2: After WO, 1 epoch of SFT on OpenHermes. The authors state that refusal ability is restored in all cases, and for instruction-tuned models, 40.5–69.8% of the refusal guardrail is restored; Qwen2.5-14B* WO-SFT's refusal rate exceeds the original baseline. The text does not give the refusal rate for that cell.
• Adversarial attacks: Section 3.5 states that relative to the WO variant, AutoDAN-Turbo ASR decreases by an average of 45.3%; the introduction states an average decrease of 37.5%. The authors state that Qwen3-4B* WO-SFT and Llama-3.1-8B* WO-SFT have ASR decreases of 32.2% and 24.4%, respectively, and are lower than their respective original aligned baselines; the paper does not specify whether these percentages are relative to WO or the original base. The per-bar values in Figure 1 are not listed in the text.
• Cyberattacks: The authors state that half the models decrease and half increase, without naming which models. The abstract states that mitigation does not significantly affect hallucination rates or natural language performance; the introduction states that hallucination rates do not noticeably increase and overall helpfulness does not noticeably decrease. The text does not give specific numbers for these two items for WO-SFT. The caption of Figure 2 is: relative to the original aligned model, lower values indicate safer or more useful, negative values indicate higher than the original model.

Other Ablations and Analyses

• The paper does not report ablations of learning rate, poisoning ratio, LoRA rank, or k.
• Section 3.1 group means: WO reduces refusal rate by 74.4% relative for instruction-tuned models and 29.3% for reasoning models; JT by 62.3% and 45.3%, respectively. This 45.3% is not the ASR reduction from SFT.
• Reasoning models have lower original refusal rates than their corresponding instruction-tuned models, e.g., Llama-3.1-8B* 45.4% vs. Llama-3.1-8B 98.4% (Table 2).
• Qwen3-4B* WO's % Decr. is 13.0, the smallest WO refusal reduction in Table 2.
• Llama-3.1-8B WO's Avg. % Incr. is −7.8% (Table 4), the only negative average hallucination increase among WO rows.
• Appendix A.2: fewshot_as_multiturn and apply_chat_template reduce performance on all models on ARC and other tasks, so those results do not use these two flags.

Limitations and Future Directions Noted by the Authors

• The authors state that WO currently requires white-box weights to estimate the refusal vector, orthogonalize, and block writing in that direction; white-box access is its only limitation to general applicability (Section 4).
• The authors state that SFT can automatically reduce WO's adversarial capability, but further reducing cyberattack effectiveness still requires additional work (Section 4).
• The authors plan to explore whether SFT plus preference alignment can suppress malicious cyber capabilities, and the role of in-distribution cyberattack preference data in that (Section 4).
• The authors plan to study black-box procedures that could achieve WO-style unalignment and call this a high-risk arms race; once done, they will notify major frontier model developers and seek containment measures (Section 4).
• The authors call for finding methods beyond SFT that can limit WO's malicious capabilities before someone bypasses WO's white-box limitation (Section 4).

Experimental Coverage

• The models under test are the 6 in Table 1: 3 instruction-tuned models and their reasoning counterparts. The claim that gpt-4o's refusal capability drops by up to 95.8% is a paraphrase of Bowen et al. (2025), not an unalignment experiment run in this paper.
• Tasks include StrongREJECT (323), AutoDAN-Turbo (HarmBench 200 targets, k=20, threshold 8.5), CyberSecEval 3 (1,000 items, 3 seeds), TruthfulQA, TofuEval (450 summaries), and the 7 helpfulness tasks in Table 5.
• AutoDAN-Turbo's victim models are only the aligned Qwen3-4B and Llama-3.1-8B; the scorer is fixed to Llama-3.1-8B.
• SFT is only done on WO models (OpenHermes, 1 epoch, learning rate 1e-5). The paper does not report doing the same SFT on JT.
• The paper does not report statistical significance tests, replication counts for AutoDAN-Turbo, or WO-SFT refusal rates, hallucination rates, helpfulness, and per-target ASR for each model.

The authors compare JT and WO on six open-source LLMs for malicious assistance capability and usefulness after removing safety alignment, and use SFT to try to reduce WO's attack gains.

• Problem: Safety alignment is meant to make models refuse harmful requests. The authors argue that a drop in refusal alone does not represent harmful assistance capability, nor whether useful capability is retained, so they compare the two unalignment methods on the same set of models.
• Method: JT inserts 2% malicious samples with jailbreak triggers into 5,000 data points and fine-tunes for 5 epochs. WO estimates the refusal direction and then orthogonally writes that direction into the weights. The WO model is then SFT'd on OpenHermes for 1 epoch. The experimental models are three instruction-tuned models and their reasoning counterparts.
• Refusal: On StrongREJECT, each method drops more on three models. Llama-3.1-8B WO has a refusal rate of 9.9% (base 98.4%, Table 2); Qwen3-4B* WO only reaches 70.6% (base 81.2%). The reasoning models' original refusal rates are lower than their corresponding instruction-tuned models.
• Attack and capability: The introduction states that WO has on average 27.7% more successful AutoDAN-Turbo attacks than JT, 40.2% more on reasoning models, and cyberattack capability is on average 6.1% higher. Qwen3-4B* WO's ASR against aligned Qwen3-4B is 88.5% (Table 3). An exception is Qwen2.5-14B's JT, which is higher than WO on both adversarial targets; JT also has six adversarial ASRs below the base model. The authors state that JT's helpfulness drops on average 13.4%, WO drops 2.2%, and WO's hallucination increase is smaller; Qwen3-4B*'s helpfulness average drop is larger for WO.
• Mitigation: The authors state that SFT restores refusal for all WO models. Adversarial ASR in Section 3.5 drops on average 45.3% relative to WO, while the introduction writes an average drop of 37.5%. Cyberattack capability drops in half and rises in half. The authors state that this mitigation does not greatly raise hallucination or lower natural language performance, but they do not give per-cell numbers.
• Authors' conclusion: The authors argue that JT makes models comply with harmful requests but is not suitable as a capable malicious assistant; WO removes safety constraints while preserving benign capability, making it more dangerous. The authors also state that JT, being poisonable, is more feasible for frontier fine-tuning interfaces; WO currently requires white-box weights. Section 4 uses the example of Claude being jailbroken to assist cyber activities against government agencies to illustrate that safety training is a threshold, and unalignment removes that threshold.