Refusal behavior in language models is mediated by a single direction

1. 01What problem does this paper attempt to solve?

   The internal mechanism of refusal behavior in chat models is unclear; the authors study whether it is mediated by a single direction and use this to construct a white-box jailbreak.

   Core question: how do chat models after safety fine-tuning represent and execute refusal behavior, and can this mechanism be simply located and removed.

   • Setting and importance: deployed chat models are fine-tuned over multiple rounds to be both helpful and harmless; the authors argue that as models gain greater autonomy, robust refusal of harmful requests is a necessary condition for safe systems (§1).
   • Insufficient existing understanding: the authors state that refusal behavior is prevalent in chat models, but its underlying mechanism remains poorly understood (abstract); prior work has studied linear representations of features such as harmlessness, but the refusal mechanism itself has not been located.
   • Core observation: the authors hypothesize that features are represented as linear directions in activation space, and verify with contrast pairs: on 13 open-source chat models (largest 72B), refusal is mediated by a one-dimensional subspace (§3).
   • Paper outputs: propose the white-box jailbreak method ORTHO based on weight orthogonalization (§4), and provide a preliminary analysis of the mechanism by which adversarial suffixes suppress the refusal direction (§5).
   • Threat model:
     • Goal: make open-source chat models no longer refuse harmful instructions while retaining other capabilities.
     • Knowledge: white-box — the attacker has access to model weights (§4).
     • Capability: directly modify model weights (rank-one modification), without requiring gradient optimization or harmful completion samples.
     • Victim systems: 13 open-source chat models, covering the Qwen, Yi, Gemma, Llama-2, and Llama-3 series (Table 1).

   Collapse this answer

2. 02What related research exists?

   Related work is divided into four categories: research on refusal mechanisms, feature direction representations, undoing safety fine-tuning, and jailbreak attacks.

   Understanding refusal in language models

   • Zheng et al. (2024) and Zou et al. (2023a): identify a harmfulness representation using harmful/harmless contrast pairs, and claim this direction differs from the refusal representation; the paper notes this is contrary to the finding in §5.1 of this paper — here the refusal direction is suppressed by adversarial suffixes.
   • Panickssery et al. (2023): extract steering vectors using contrastive multiple-choice completions; refusal can be modulated in multiple-choice settings but is ineffective in long-form generation.
   • Wei et al. (2024): removing safety-critical neurons and ranks can weaken safety mechanisms while preserving utility; Wang and Shu (2024) propose the Trojan Activation Attack, which adds steering vectors at inference time to bypass refusal.

   Features as directions

   • Extracting feature directions from contrast pairs is an established technique (Burns et al., 2022; Panickssery et al., 2023; Zou et al., 2023a); adding feature vectors to the residual stream can change behavior (Li et al., 2024a; Marks and Tegmark, 2023; Tigges et al., 2023; Turner et al., 2023).
   • Multiple works show that directions in activation space have more "feature-like" properties than neurons (Bolukbasi et al., 2016; Elhage et al., 2022; Park et al., 2023b); sparse autoencoders can discover feature directions unsupervised (Bricken et al., 2023; Templeton et al., 2024).

   Undoing safety fine-tuning

   • Lermen et al. (2023), Yang et al. (2023), Zhan et al. (2023): fine-tuning on malicious samples suffices to remove safety guardrails with little capability loss; the paper notes that fine-tuning requires harmful instructions and completion samples, whereas this paper's method requires only harmful instructions.
   • Mechanistic interpretability work such as Lee et al. (2024): fine-tuning does not change the relevant internal circuits; for example, detoxification fine-tuning can be undone by scaling a small number of MLP weights.

   Jailbreaks

   • Social-engineering jailbreaks (Perez et al., 2022; Shah et al., 2023b; Wei et al., 2023); optimized adversarial suffixes (Zou et al., 2023b; Andriushchenko et al., 2024; Liao and Sun, 2024). The paper notes that this paper's method does not require modifying the input prompt, but requires weight access.

   Baselines and benchmarks: on the 159 standard behaviors of HarmBench (Mazeika et al., 2024), compare ASR against GCG, GCG-M, GCG-T, AP, PAIR, HUMAN (human templates), and DR (direct response) baselines (§4.2, §F.1). The authors position this paper as: using internal model understanding to both explain existing vulnerabilities and discover new ones, with a method simpler than fine-tuning-based jailbreaks.

   Collapse this answer

3. 03How does the paper address this problem?

   Extract the refusal direction using difference-in-means, intervene via direction ablation/activation addition and subtraction, and give an equivalent weight orthogonalization implementation.

   Overall approach: find a single "refusal direction" in residual stream activations, verify its necessity (ablation) and sufficiency (addition) for refusal behavior, then convert inference-time ablation into an equivalent weight modification. The method's core is difference-in-means direction extraction and directional ablation.

   Data and setup

   • Datasets: Dharmful is taken from AdvBench, MaliciousInstruct, TDC2023, HarmBench; Dharmless is taken from Alpaca. Each contains 128 training and 32 validation samples, and is pairwise disjoint from the evaluation sets (§2.2, §A).
   • Analysis region: only activations at post-instruction token positions (the template tokens after the instruction in the chat template) are analyzed (§2.1).

   Direction extraction (§2.3)

   • difference-in-means: for each layer l and each post-instruction position i, compute the difference between the mean activations of harmful and harmless prompts r_i^(l) = μ_i^(l) − ν_i^(l), yielding |I|×L candidate vectors.
   • Selecting a single vector: on the validation set, evaluate each candidate's bypass_score (mean refusal metric after ablation), induce_score (mean refusal metric after addition) and kl_score (KL divergence of the last-token distribution on harmless data before and after ablation), and select the vector with the smallest bypass_score that satisfies induce_score>0, kl_score<0.1, l<0.8L (§C.1).

   Intervention methods (§2.4)

   • Activation addition: x^(l) ← x^(l) + r^(l), added only at all token positions of the extraction layer l, used to induce refusal.
   • Direction ablation: x′ ← x − r̂r̂^⊤ x, performed at all layers and all token positions, preventing the model from representing that direction.

   Weight orthogonalization (§4.1)

   • Orthogonalize the columns of all matrices that write to the residual stream (embedding, positional embedding, attention output, MLP output matrices and output biases) with respect to r̂: Wout ← Wout − r̂r̂ᵀWout.
   • The paper proves in §E that this weight modification is equivalent to inference-time direction ablation, so the results in §3.1 also characterize the effect of the weight modification.

   Evaluation methods (§2.5)

   • refusal score: 1 if the completion contains any "refusal substring" (string matching, §D.1).
   • safety score: use Meta Llama Guard 2 to classify completions as safe/unsafe (§D.2).
   • Generation settings: greedy decoding, maximum generation length 512 tokens (§2.5).
   • ASR: attack success rate on the HarmBench test set as judged by its provided classifier (§4.2).

   Collapse this answer

4. 04What experiments did the paper conduct?

   Validate the effect of direction ablation/addition on 13 models, compare jailbreak baselines on HarmBench, and evaluate capability retention and the adversarial suffix mechanism.

   Experimental setup

   • Tested models: 13 open-source chat models — Qwen 1.8B/7B/14B/72B Chat, Yi 6B/34B Chat, Gemma 2B/7B IT, Llama-2 7B/13B/70B Chat, Llama-3 8B/70B Instruct (Table 1).
   • Evaluation sets: §3 uses JailbreakBench (100 harmful instructions, 10 categories) and 100 Alpaca harmless instructions; §4 uses the HarmBench test set of 159 standard behaviors (6 categories) (§A).
   • Metrics: refusal score (refusal substring matching), safety score (Llama Guard 2 classification), ASR (HarmBench classifier); capabilities use MMLU, ARC, GSM8K, TruthfulQA, etc.
   • Baseline jailbreaks: GCG, GCG-M, GCG-T, AP, PAIR, HUMAN, DR (§F.1).
   • Sample size: n=100 in Figure 1 and Figure 3, error bars are standard error (§D.4); §5 analysis uses 128 instructions (§H.2).

   Main results

   Figure 1 shows direction ablation lowers refusal score and safety score on all 13 models (specific values are presented as bar heights in the figure and cannot be read out one by one in text conversion); Figure 3 shows activation addition makes models refuse harmless instructions. HarmBench ASR comparisons are as follows (from Table 2, values in parentheses are ASR without system prompt):

   Swipe horizontally to read the table

   Model	ORTHO	GCG-M	GCG-T	DR	GCG	PAIR
   Llama-2 7B	22.6 (79.9)	20.0	16.8	0.0	34.5	7.5
   Llama-2 13B	6.9 (61.0)	8.7	13.0	0.5	28.0	15.0
   Llama-2 70B	4.4 (62.9)	5.5	15.2	0.0	36.0	7.5
   Qwen 7B	79.2 (74.8)	73.3	48.4	7.0	79.5	58.0
   Qwen 14B	84.3 (74.8)	75.5	46.0	9.5	83.5	51.5
   Qwen 72B	78.0 (79.2)	-	36.6	8.5	-	54.5

   • The authors state that ORTHO as a universal jailbreak is comparable to per-prompt optimized GCG on the Qwen series (§4.2).
   • The effect of system prompts differs by model family: Llama-2 ASR drops substantially with system prompts, while Qwen is almost unchanged; the analysis in §F.2 shows this difference cannot be explained by prompt content alone.

   Capability retention evaluation

   • Setup: run MMLU, ARC, GSM8K, TruthfulQA on orthogonalized models (LM Evaluation Harness, settings consistent with the Open LLM Leaderboard); §G.1 additionally has WinoGrande and TinyHellaSwag (§4.3).
   • Results: except for Qwen 7B and Yi 34B, all metrics fall within the 99% confidence interval of the original models; in Table 3, MMLU/ARC/GSM8K changes for the largest model in each family range from -0.7 to +1.5. TruthfulQA consistently decreases (e.g., Llama-3 70B drops from 61.8 to 59.5, Table 3).
   • Authors' interpretation: the authors consider that TruthfulQA involves categories such as misinformation, stereotypes, and conspiracies that are close to the refusal domain, so the behavioral difference is intuitively reasonable (§4.3, §G.2).

   Adversarial suffix mechanism analysis

   • Setup: study a single GCG-generated adversarial suffix on Qwen 1.8B Chat, comparing three conditions — no suffix / adversarial suffix / random suffix — on 128 harmful instructions (§5.1, §H.1).
   • Results: Figure 5 shows the last-token activation of harmful instructions has a high cosine similarity with the refusal direction; adding a random suffix keeps it high, while adding the adversarial suffix suppresses it to a level close to harmless instructions. Figure 6(a) shows the outputs of the 8 attention heads with the highest DFA on the refusal direction are suppressed; Figure 6(b) shows the attention of these heads shifts from the instruction region to the suffix region.
   • Authors' interpretation: the authors state that the adversarial suffix "hijacks" the attention of important heads, moving it away from the harmful instruction content (§5.2).

   Other ablations and analyses

   • CE loss (Table 9): direction ablation yields CE loss close to baseline on The Pile, Alpaca, and on-distribution data; activation addition (subtracting the vector) yields noticeably higher CE loss, on which basis the authors call ablation more "surgical" than activation addition (§G.3).
   • Comparison with fine-tuning (Table 10): on Llama-3 8B, after direction ablation the refusal score is 0.01 and the safety score is 0.15; after LoRA fine-tuning (rank=16, 4 epochs) they are 0.00 and 0.08; the authors note that fine-tuning requires constructing a harmful completion dataset, whereas this paper's method requires only harmful instructions (§I.2).
   • Base models (Figure 24): the base models of Qwen 1.8B, Gemma 7B, Llama-3 8B, and Qwen 14B also highly express the refusal direction on harmful prompts; the authors consider that this direction already exists before safety fine-tuning and is "reused" during fine-tuning (§J).
   • Behavior of orthogonalized models: §L.1 shows orthogonalized models can still output the standard refusal string when explicitly asked; §L.3 shows their meta-level explanations of their own new refusal behavior are incoherent.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list five limitations, including generalization, the heuristic nature of the extraction method, the scope of the suffix analysis, the coherence metric, and the semantics of the direction.

   Limitations and future directions noted by the authors

   • Generalization: results may not generalize to untested models, especially larger current proprietary models and future models (§7 Limitations).
   • Extraction method: the direction extraction method may not be optimal and relies on several heuristics; the authors view this paper as a proof of existence and leave method improvements to future work (§7).
   • Scope of suffix analysis: the analysis of adversarial suffixes does not provide a complete mechanistic understanding and is limited to a single model and a single adversarial example (§7).
   • Coherence metrics: coherence of chat models is difficult to measure; the authors consider each metric used to be flawed, so they use multiple metrics to give a broad view (§7).
   • Semantics of the direction: the semantic meaning of the direction is unclear; it may represent concepts such as "harm" or "danger", or may not be directly semantically interpretable (§7).
   • System prompts: the relationship between system prompts and refusal behavior requires more systematic study, which the authors leave as a future direction (§F.2).

   Scope of experimental coverage

   • The tested models are 13 open-source chat models, with parameter sizes from 1.8B to 72B, covering two alignment approaches, AFT and APO (Table 1).
   • Jailbreak comparisons are conducted only on 6 models in HarmBench (three Llama-2, three Qwen); Qwen 72B lacks the GCG-M, GCG, and AP columns (Table 2).
   • The adversarial suffix mechanism analysis is conducted only on Qwen 1.8B Chat and a single suffix (§H.1).
   • Base model comparisons cover only four models: Qwen 1.8B, Gemma 7B, Llama-3 8B, and Qwen 14B (§J).
   • The paper does not report human agreement validation for refusal/safety evaluation; refusal score is string matching, and safety score is judged by Llama Guard 2 (§2.5).

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors state that refusal is mediated by a single direction, that ablating it enables a white-box jailbreak with little capability loss, and they provide a preliminary explanation of how adversarial suffixes work.

   A work combining mechanistic interpretability and jailbreaking: studies how refusal behavior in safety-fine-tuned chat models is represented, and uses this understanding to construct an attack.

   • Problem: chat models generally refuse harmful requests, but the mechanism is unclear; the authors aim to locate the refusal mechanism and evaluate its fragility.
   • Method: compute a difference-in-means direction using contrast pairs of 128 harmful and 128 harmless instructions, and obtain a single "refusal direction" through heuristic selection; direction ablation bypasses refusal, activation addition induces refusal; weight orthogonalization (ORTHO) converts ablation into an equivalent rank-one weight modification, requiring no gradients or harmful completion samples.
   • Main results:
     • On 13 open-source models (largest 72B), ablating a single direction lowers the refusal rate and produces unsafe completions (Figure 1); adding the direction makes models refuse harmless instructions (Figure 3).
     • ORTHO's ASR on HarmBench: Qwen 14B is 84.3%, Qwen 7B is 79.2%; the Llama-2 series is only 4.4–22.6% with system prompts, rising to 61.0–79.9% without them (Table 2).
     • Capability evaluations mostly change very little, but TruthfulQA consistently decreases (e.g., Llama-3 70B drops 2.3, Table 3); Qwen 7B shows statistically distinguishable decreases on MMLU and GSM8K (§G.1).
     • Mechanism analysis shows adversarial suffixes suppress the expression of the refusal direction and "hijack" the attention of key attention heads from the instruction region to the suffix region (Figure 5, 6).
   • Authors' conclusion: the authors consider current safety fine-tuning methods fragile, with a simple rank-one weight modification nearly eliminating refusal behavior, and hope to contribute to the discussion on responsible release of open-source models (§1, §7).

   Collapse this answer

Related papers