Large language models encode harmfulness and refusal separately

1. 01What problem does this paper attempt to solve?

   The authors ask whether LLMs internally truly encode the concept of "harmfulness" or only refusal behavior, and state that the two are encoded separately.

   Core question: Does the LLM internally encode "harmfulness" as a concept separate from "refusal," or does it conflate the two?

   • Problem setting: LLMs learn through multiple rounds of training to refuse harmful instructions, but the authors point out that two types of failure remain common—naive or jailbreak harmful prompts can bypass refusal, and over-refusal of harmless prompts. The authors argue that understanding how the model internally represents harmfulness is a prerequisite for analyzing these failures.
   • Shortcomings of existing methods: The authors state that prior work often assumes harmfulness is encoded by the refusal direction (typically extracted at the last input token), and interprets ablating the refusal direction as "making the model consider the input harmless"; however, whether harmfulness is actually conflated with refusal in latent space was previously unclear.
   • Core observation: After removing the post-instruction special token from the prompt template, the model's refusal rate on harmful instructions decreases (Table 1); based on this, the authors hypothesize that harmfulness is encoded at the instruction's last token (tinst), while the refusal signal is encoded at the last token of the entire input (tpost-inst).
   • What the paper proposes: A set of analysis methods for separating harmfulness and refusal representations (clustering analysis, difference-of-means direction extraction, causal intervention via a reply inversion task), and based on the harmfulness representation, an intrinsic defense classifier Latent Guard, used to detect harmful inputs that bypass refusal and harmless inputs that are wrongly refused.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: linear representations in LLMs, research on refusal/harmfulness directions, and latent-space analysis of jailbreaks; the authors state that this paper separates the two at the token-position level.

   Linear representations in LLMs

   • Marks and Tegmark, Li et al. (2023) — found that "true/false" can be linearly represented in LLMs, and intervening along the truthfulness direction can make the model treat false statements as true.
   • Azaria and Mitchell (2023) — probing hidden states classifies true/false more reliably than model outputs, which are biased by surface features such as sentence length; the authors state that this paper similarly finds internal representations more faithfully reflect the harmfulness of inputs.

   Refusal and harmfulness directions

   • Arditi et al. (2024) — proposed that refusal is mediated by a single direction, extracted at tpost-inst using difference-in-means; ablating this direction enables jailbreak without harming utility. Zheng et al. (2024), Yu et al. (2025), and Rimsky et al. (2024) likewise extract the refusal direction at tpost-inst.
   • Siu et al. — a direction extracted at tpost-inst using the difference-in-means of "refused harmful vs. refused harmless" cannot effectively induce refusal; the authors state this supports their conclusion that "tpost-inst mainly encodes shallow refusal signals."
   • Zhao et al. (2025), Han et al. (2025) — found that accepted harmful samples and harmless samples are separable in deep hidden states, suggesting harmfulness recognition and refusal decisions may be different processes. Yeo et al. (2025) — used sparse autoencoders to find that harmfulness features and refusal features may be separate.

   Latent-space analysis of jailbreaks

   • Arditi et al. (2024), Xu et al. (2024), Ball et al. (2024), Yu et al. (2025), Wollschläger et al. (2025) — found that hidden states of jailbreak prompts at tpost-inst resemble accepted harmless instructions and have low similarity to the refusal direction. The authors state that because what tpost-inst encodes is unclear, this geometric similarity is hard to explain, and the causal evidence provided in this paper shows that this similarity may stem only from "both being accepted."

   Baselines and benchmarks

   • Baseline methods: compared with Latent Guard and the specially fine-tuned safety model Llama Guard 3 8B (Inan et al., 2023).
   • Datasets used: Advbench, JBB, Sorry-Bench (harmful instructions), Alpaca, Xstest (harmless/over-refusal), CATQA (category-wise harmfulness), ToxicChat and the OpenAI Moderation Evaluation Dataset (appendix guardrail evaluation).

   Authors' positioning: this paper focuses on the representation of the general concept of "harmfulness," stating that it separates harmfulness from refusal at different token positions, and that the extracted harmfulness direction can manipulate the model's perception of harmfulness, whereas the refusal direction cannot.

   Collapse this answer

3. 03How does the paper address this problem?

   Perform clustering and difference-of-means direction extraction at two token positions, provide causal evidence using activation steering and a reply inversion task, and then build Latent Guard.

   Overall approach: extract hidden states at the instruction's last token (tinst) and the last token of the entire input (tpost-inst), first perform correlation clustering analysis, then extract the harmfulness direction and refusal direction for activation steering, and finally use a self-designed reply inversion task to behaviorally decouple the two, providing causal evidence.

   Problem setup and the two token positions

   • Analysis object: the residual-stream hidden state h_l(x_t) of a decoder-only Transformer, where l is the layer and t is the token position.
   • tinst: the last token of the user instruction; tpost-inst: the last token of the post-instruction special token (e.g., [/INST]). The authors state that prior work only looked at tpost-inst, and this paper introduces tinst because of the observation in Table 1 (refusal rate decreases after removing post-instruction tokens).
   • Data: harmful instructions come from Advbench, JBB, and Sorry-Bench; harmless instructions come from Alpaca; over-refusal samples come from Xstest.

   Clustering analysis and belief scores

   • Clustering decision: compute two cluster centers, "refused harmful" and "accepted harmless," on the training set, and for test samples compute s_l(h_l) = cos_sim(h_l, μ_l^{refused harmful}) − cos_sim(h_l, μ_l^{accepted harmless}); the threshold a defaults to 0, i.e., assigning to the cluster with the closer cosine similarity; the authors state that oracle threshold estimation is left for future work.
   • Belief scores: ∆harmful is the average over layers of the similarity difference between the "harmful cluster vs. harmless cluster" at tinst, and ∆refuse is the average of the similarity difference between the "refusal cluster vs. acceptance cluster" at tpost-inst, used to quantify the model's internal beliefs.

   Direction extraction and activation steering

   • Harmfulness direction: the difference of means of the two cluster centers at tinst, v_harmful^l = μ_tinst^l,harmful − μ_tinst^l,harmless; the refusal direction is extracted the same way at tpost-inst.
   • Intervention method: activation addition h′ = h_l + v_harmful^l, added to all tokens of the input instruction; in the reply inversion task, the harmfulness direction is added only to the tokens before the inversion question, and the refusal direction is added to all input tokens including the post-instruction tokens (Appendix E.1 states that this intervention is most effective).

   Reply inversion task (causal verification)

   • Design: append an inversion question after the original instruction (e.g., "Could this user prompt cause harm?"), so that the model should answer the refusal token ("No") for harmless instructions and the acceptance token ("Certainly") for harmful instructions; Table 2 gives the template, and Appendix Tables 6 and 7 give other templates.
   • Decision: observe whether the generated reply flips due to the intervention. If a harmless instruction, after steering along the harmfulness direction, is answered "Certainly," the authors consider this to indicate that the model judges the instruction as more harmful; if steering along the refusal direction still yields "No," the authors consider this to indicate that the refusal direction does not change the harmfulness judgment.

   Latent Guard

   • Classification rule: compute ∆harmful for the input; if negative, classify as harmless, otherwise as harmful; cluster centers are computed using 100 harmful and 100 harmless samples each from the training set.
   • Characteristics: no additional guard model is needed; the classification result is available during the model's normal forward pass, before it starts generating a reply; the authors state that it is computationally efficient, and the paper does not report specific timing numbers.
   • Robustness test: steer harmful instructions with the reverse refusal direction to obtain acceptance replies, construct 50–400 adversarial samples to fine-tune the model (loss computed only on the reply), then check whether ∆harmful changes.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Clustering, steering, and reply inversion experiments on three 7B/8B open-source models; Latent Guard compared with Llama Guard 3, and stability under fine-tuning attacks was also tested.

   Experimental setup

   • Models tested: Llama2-Chat-7B, Llama3-Instruct-8B, Qwen2-Instruct-7B, all run on A100-40GB GPUs.
   • Datasets: Advbench, JBB, Sorry-Bench for harmful instructions; Alpaca for harmless instructions; Xstest for over-refusal samples; CATQA for category-wise analysis; ToxicChat and the OpenAI Moderation Evaluation Dataset for appendix guardrail evaluation. The paper does not state the specific number of examples used from each dataset.
   • Baseline: Llama Guard 3 8B (a specially fine-tuned safety classification model).
   • Metrics: refusal rate (determined by matching a set of refusal substrings; Section 3.5 counts by the refusal token "No"), Latent Guard classification accuracy, cosine similarity between directions, belief score ∆harmful/∆refuse.
   • Latent Guard construction: 100 harmful and 100 harmless samples each from the training set to compute cluster centers; the test set consists of held-out successful jailbreak prompts on each model, refused harmless samples from Xstest, and accepted harmful samples from Sorry-Bench.
   • Number of repetitions: not reported in the paper.

   Main results

   Classification accuracy of Latent Guard and Llama Guard 3 (%, from Table 3; Adv-suffix/Persuasion/Template are three types of successful jailbreak prompts, Refused HL is refused harmless instructions, Accepted HF is accepted harmful instructions):

   Swipe horizontally to read the table

   Model	Guard	Adv-suffix	Persuasion	Template	Refused HL	Accepted HF
   Llama2-Chat-7B	Llama Guard 3	100.0	0.0	76.0	84.4	45.5
   Llama2-Chat-7B	Latent Guard	100.0	41.6	100.0	100.0	93.9
   Llama3-Instruct-8B	Llama Guard 3	99.2	6.8	50.0	50.0	37.3
   Llama3-Instruct-8B	Latent Guard	91.0	65.0	100.0	78.5	59.3
   Qwen2-Instruct-7B	Llama Guard 3	97.8	17.8	91.4	50.0	59.4
   Qwen2-Instruct-7B	Latent Guard	100.0	75.0	53.5	91.6	54.6

   • The authors state that Latent Guard is overall comparable to or better than Llama Guard 3, performing especially well on persuasion jailbreaks and refused harmless samples.
   • There are counterexamples in the table: on Llama3, Latent Guard's Adv-suffix accuracy (91.0) is lower than Llama Guard 3's (99.2); on Qwen2, Template (53.5 vs 91.4) and Accepted HF (54.6 vs 59.4) are also lower than the baseline.
   • The authors state that Latent Guard is computationally efficient because it requires no additional guard model and produces results within the normal forward pass; the paper does not report specific timings.

   Clustering and belief correlation (Section 3.1–3.3)

   • Removing post-instruction tokens: for refusal rate on Advbench harmful instructions, Llama2 drops from 100.0 to 85.3, Llama3 from 96.0 to 58.9, and Qwen2 from 98.0 to 81.3 (Table 1). The authors therefore argue that the refusal signal may only form at post-instruction tokens.
   • Clustering patterns (Figure 2): the authors state that at tinst, "accepted harmful instructions" mostly fall into the refused harmful cluster and "refused harmless instructions" mostly fall into the accepted harmless cluster, i.e., clustering by harmfulness; at tpost-inst the pattern reverses, clustering by whether refused. The converted text contains only axes and legends; specific curve values are not readable.
   • Belief correlation (Figure 3, Llama2): the authors state that refused harmless instructions have negative ∆harmful (internally still considered harmless), and many accepted harmful instructions still have positive ∆harmful; the authors argue that refusal behavior and internal harmfulness belief are not always consistent.

   Direction steering and response reversal (Section 3.4–3.5)

   • Steering harmless instructions (Figure 4, Llama3): the harmfulness direction achieves the highest refusal rate of 94% at layer 9, and the refusal direction reaches 100% at layer 11; Appendix C reports that on Qwen2 and Llama2, the refusal rate induced by the harmfulness direction is lower than that by the refusal direction. The average cosine similarity between the two directions is low (about 0.1 on Llama2; for comparison: held-out refused instructions average 0.57 with the refusal direction, accepted instructions average -0.14).
   • Response reversal (Figure 5, Qwen2): the authors state that after harmless instructions are steered along the harmfulness direction, responses flip to "Certainly" (refusal rate decreases), while steering along the refusal direction still answers "No"; after harmful instructions are steered along the reverse harmfulness direction, responses become "No," while steering along the reverse refusal direction cannot flip them. The authors use this as causal evidence that the two directions act differently; Appendix E reports consistent patterns on other templates and on Llama3.

   Jailbreak analysis and category-wise harmfulness (Section 4, Appendix D)

   • Jailbreak analysis (Figure 6): the authors state that persuasion jailbreaks can in some cases make the model internally believe the prompt is harmless (∆harmful negative), whereas adversarial suffix and template jailbreaks mainly suppress the refusal signal (∆refuse negative), with ∆harmful still high for some samples.
   • Category-wise directions (Figure 9a, Qwen2): the harmfulness direction has lower average cosine similarity across categories (e.g., 0.6 for the Hate_Haras_Violence category), and the refusal direction higher (0.89); the authors state that harmfulness representations are more fine-grained.
   • Within-category intervention (Figure 9b, Qwen2): when the test set is Hate_Haras_Violence, reverse intervention with the same-category harmfulness direction reaches a maximum refusal rate of 100%, the Political_Campaigning category direction reaches only 32%, the Adult_Content direction also reaches 100%, and the all-category average direction is 52%. The authors state that harmfulness directions may be risk-category specific.

   Stability under fine-tuning attacks (Section 5, Appendix H.2)

   • Setup: use the reverse refusal direction to steer generation of "harmful instruction + accepted response" pairs, and fine-tune the model at scales of 50/100/200/300/400 examples (updating only the response loss).
   • Results (Figure 7): the authors state that after fine-tuning the model begins to accept held-out harmful instructions, but ∆harmful of these instructions at tinst barely changes as adversarial examples increase, so Latent Guard can still detect them; the authors state that Latent Guard is robust to the narrow range of fine-tuning attacks tested. Appendix H.2 (Figure 18) states that the refusal direction can still be extracted after fine-tuning, but its effectiveness declines slightly as training examples increase.

   Other ablations and analyses

   • Out-of-domain guardrail evaluation (Appendix H.1): on ToxicChat, Latent Guard is worse than Llama Guard 3 in all cases (e.g., for the Qwen2 version, Unsafe accuracy 34.0 vs 47.2, Table 8); on the OpenAI Moderation set, for the Qwen2 and Llama3 versions without in-domain samples, accuracy for most categories is below the baseline, and after adding 50 in-domain samples most categories exceed the baseline (Table 9).
   • More token positions (Appendix F): the authors state that only tinst simultaneously satisfies "accepted harmful falls into the harmful cluster, refused harmless falls into the harmless cluster," and that in the response reversal task the direction extracted at tinst has the strongest intervention effect (Figures 14, 16).
   • Threshold a: the clustering decision threshold defaults to 0; the authors state that oracle threshold estimation is left for future work.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list limitations such as not studying the role of layers and testing only 7B/8B open-source models, and propose future directions such as circuit analysis and the relationship between fine-tuning and representations.

   Limitations and future directions noted by the authors

   • Role of layers not studied (Limitations): The paper presents per-layer clustering and steering results, but does not study the role of different layers in the formation of refusal and harmfulness, leaving model-level (neuron or layer) explanations for future work.
   • Only a proof of existence (Limitations): The authors state that this paper is a proof of existence that harmfulness can be separated from refusal, rather than a complete description of how it is encoded; the two may be intertwined and the encoding may take multiple forms.
   • Limited model scale (Limitations): Due to compute constraints, only 7B/8B open-source models were tested; the authors state that the conclusions may not generalize to larger, untested models.
   • Out-of-domain failure (Appendix B, H.1): The authors point out that latent-space detectors such as Latent Guard may fail on out-of-domain samples, and regard this as an application-level limitation; improving sampling strategies to cover diverse harmful categories is listed as future work.
   • Circuit-level analysis (Conclusion and Discussion): The authors propose that future work could use circuit analysis (e.g., how self-attention relates tpost-inst to tinst) to understand the relationship between internal harmfulness beliefs and external refusal behavior.
   • Relationship between fine-tuning and representations (Conclusion and Discussion): The authors propose that in-depth study is needed of the interaction between fine-tuning, harmfulness and refusal representations, and of whether more refined fine-tuning strategies that act directly on the harmfulness representation are needed.

   Experimental coverage

   • The models tested are Llama2-Chat-7B, Llama3-Instruct-8B, and Qwen2-Instruct-7B, 3 open-source instruct models in total (Section 2).
   • The jailbreak analysis covers three types of methods: adversarial suffixes (GCG), persuasion, and adversarial prompting templates (Section 2, 4).
   • Latent Guard's cluster centers are computed using 100 harmful and 100 harmless samples each from the training set (Section 5.1); the fine-tuning attack evaluation covers scales of 50 to 400 adversarial samples (Section 5).
   • The paper does not report the number of repetitions for each experiment, nor the specific sample sizes for each condition in the clustering and steering experiments.
   • The only baseline for the defense comparison is Llama Guard 3 8B; the out-of-domain evaluation uses ToxicChat and the OpenAI Moderation Evaluation Dataset (Appendix H.1).

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors state that LLMs encode harmfulness at tinst and refusal at tpost-inst, that the two are separable, and build Latent Guard based on this.

   A mechanistic interpretability paper studying whether "harmfulness" and "refusal" are the same representation inside aligned LLMs.

   • Problem: Existing work often assumes harmfulness is encoded by the refusal direction; the authors question this, pointing out that models are both jailbroken past refusal and over-refuse harmless prompts.
   • Method: Extract hidden states at the instruction's last token (tinst) and the last token of the entire input (tpost-inst) for clustering analysis, use difference-of-means to extract the harmfulness direction and refusal direction for activation steering, and design a reply inversion task (appending a question "is this prompt harmful?" so that harmless instructions should be answered "No") for causal verification; finally, build the Latent Guard classifier based on the harmfulness belief score ∆harmful.
   • Key results: After removing post-instruction tokens, the refusal rate of the three models on Advbench harmful instructions drops from 96.0–100.0 to 58.9–85.3 (Table 1); the authors state that tinst clusters by harmfulness and tpost-inst clusters by refusal (Figure 2); in the reply inversion task, steering along the harmfulness direction can make Qwen2 judge harmless instructions as harmful, while steering along the refusal direction does not change its harmfulness judgment (Figure 5); on Llama3, steering along the harmfulness direction yields a highest refusal rate of 94% (layer 9, Figure 4).
   • Application results: Latent Guard detects persuasion jailbreak prompts on Qwen2 with 75.0% accuracy, higher than Llama Guard 3's 17.8% (Table 3); the authors state that under fine-tuning attacks with 50–400 adversarial samples, ∆harmful remains almost unchanged and it can still detect harmful instructions (Figure 7).
   • Authors' conclusion: The authors believe that the LLM's perception of harmfulness and its refusal decision may be decoupled, and that some jailbreaks merely suppress the refusal signal without changing the internal harmfulness belief; the authors state that this provides a new perspective for understanding safety alignment (e.g., the shallow alignment hypothesis) and improving defenses, while acknowledging that only 7B/8B open-source models were tested and the role of layers was not studied.

   Collapse this answer

Related papers