Skip to content

Refusal Register Can Be Causally Manipulated: Decomposing the Refusal Direction of Gemma-2-9B-IT into Category-Aligned Components

Refusal Register is Causally Steerable: Decomposing the Refusal Direction into Category-Aligned Components in Gemma-2-9B-IT

arXiv: 其他 AI 安全论文· arXiv:2611.20405· Janhavi Khindkar, Nataliia Povarova·Indexed by this site · 2 days ago· Originally publishedAlignment· Its place in the scan record →

Quick read

Interpretability

Based on the paper PDF (AI-generated); refer to the original

The authors find two class-aligned refusal directions at layer 20 of Gemma-2-9B-IT; injecting the harmful-class direction can rewrite self-referential-class refusals into a content style while maintaining refusal.

Questions
Existing work has shown that refusal can be mediated by a single direction or multiple directions; the authors ask a different question: whether refusal-related directions also encode the register of refusal (i.e., how to refuse), and not merely refuse versus not refuse.
Methods
At layer 20 of Gemma-2-9B-IT, linear SVMs yield two directions, B1 (harmful vs neutral) and B2 (self-referential vs neutral); combined with Gemma Scope SAE feature ablation, feature-clamping steering, and DoM direction injection, the causal effects on refusal rate, compliance, and refusal word frequency are measured.
Experiments and results
B1 and B2 cosine 0.252; ablation each selectively reduces refusal of its own category; injecting the DoM direction corresponding to B1 into SELFREF prompts raises content-style word frequency from 2.55 to 10.37/100 token (α=2.0), and the model still refuses but switches to a content style, while B2's reverse effect is about an order of magnitude smaller; the 2B model replicates qualitatively.
Limitations and future work
Limitations noted by the authors: single model family, few prompts for fitting and evaluation (10 FIT, 10–20 EVAL), only two refusal categories measured, register measured with a regex word list, layer-wise analysis covering only 3 of 42 layers; future directions include cross-family replication and denser layer sweeps.
Experimental setupGemma-2-9B-IT、Gemma-2-2B-IT · 自建提示词集:HARMFUL、SELFREF、NORMAL 三类(每类 10 条 FIT、10–20 条 EVAL) · refusal-rate、compliance level (0–5) et al.
Evaluated models
Gemma-2-9B-ITGemma-2-2B-IT
Benchmarks
自建提示词集:HARMFUL、SELFREF、NORMAL 三类(每类 10 条 FIT、10–20 条 EVAL)
Metrics
refusal-ratecompliance level (0–5)vocabulary rate (hits per 100 tokens)
AI summaryThe researchers find that refusal-related directions in Gemma-2-9B-IT not only encode whether the model refuses, but also carry information about refusal wording style, and that this style can be causally manipulated. At layer 20, they obtain two linear directions with an SVM: B1 distinguishes harmful from neutral content, and B2 distinguishes self-referential from neutral content. After injecting the mean-difference direction of harmful versus neutral at alpha=2.0 into the residual stream of self-referential prompts, content-safety word hits rise from 2.55 per 100 tokens to 10.37, while magnitude-matched random-direction controls give 1.13-2.16; the model still refuses, but the wording shifts from a “as a language model” style to a “this would be harmful” style. The effect is qualitatively reproduced in Gemma-2-2B-IT at layer 12 (1.72 rising to 6.37, random control 1.35), with a narrower effective injection window than 9B. The causal effect of B1 is consistently larger than that of B2 across ablation, sufficiency and injection experiments; the authors suggest that B1 encodes both category detection and refusal-vocabulary information, while B2 mainly encodes category detection.

The researchers find that refusal-related directions in Gemma-2-9B-IT not only encode whether the model refuses, but also carry information about refusal wording style, and that this style can be causally manipulated. At layer 20, they obtain two linear directions with an SVM: B1 distinguishes harmful from neutral content, and B2 distinguishes self-referential from neutral content. After injecting the mean-difference direction of harmful versus neutral at alpha=2.0 into the residual stream of self-referential prompts, content-safety word hits rise from 2.55 per 100 tokens to 10.37, while magnitude-matched random-direction controls give 1.13-2.16; the model still refuses, but the wording shifts from a “as a language model” style to a “this would be harmful” style. The effect is qualitatively reproduced in Gemma-2-2B-IT at layer 12 (1.72 rising to 6.37, random control 1.35), with a narrower effective injection window than 9B. The causal effect of B1 is consistently larger than that of B2 across ablation, sufficiency and injection experiments; the authors suggest that B1 encodes both category detection and refusal-vocabulary information, while B2 mainly encodes category detection.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors ask: do refusal directions encode the register of refusal (how to refuse), and not merely refuse versus not refuse.

    Core question: do refusal-related linear directions carry information about the refusal register (i.e., with what wording and framing the model refuses), and not merely determine whether refusal occurs.

    • Background: Arditi et al. (2024) found that a single residual stream direction mediates refusal; Wollschläger et al. (2025) found multiple refusal directions and concept cones; the authors consider these works to answer "whether refusal occurs."
    • Gap in existing work: the authors state that Joad et al. (2026) extracted refusal directions for 11 categories and found that differences between directions are mainly reflected in refusal style, but did not study self-referential (system-prompt leakage) prompts, nor perform dose-response-style register transfer measurements.
    • Core observation: the model refuses harmful content requests and self-referential probes in markedly different styles; the authors hypothesize that these two types of refusal are mediated by different directions, and that the directions carry register information.
    • Paper's approach: at layer 20 of Gemma-2-9B-IT, extract two SVM directions B1 (HARMFUL vs NORMAL) and B2 (SELFREF vs NORMAL), and through three types of intervention—SAE feature ablation, feature-clamping steering, and DoM direction injection—test whether the causal roles of the two directions are separable and whether they are asymmetric.
    • Positioning: the authors explicitly state that the contribution is not "multiple category directions exist" or "directions differ in style" (these are established by prior work), but rather the randomized controlled dose-response measurement of register transfer for the self-referential category, the SAE ablation and sufficiency results for the two categories, and the consistent asymmetry between the two directions.
  2. What related research exists?

    Developed along the refusal-direction line of research: single direction → multiple directions → categorization; this paper adds a measurement of stylistic transfer for the self-referential category.

    Refusal directions

    • Arditi et al. (2024)—found that a single residual stream direction mediates harmful-content refusal in instruction-tuned models; ablating this direction bypasses safety training, and steering along it induces refusal.
    • Wollschläger et al. (2025)—proposed that refusal is mediated by multiple independent directions and a multi-dimensional concept cone, and proposed a stricter-than-orthogonality criterion of "representation independence"; Zhao et al. (2025)—found that harmfulness and refusal are encoded at different token positions.
    • Prakash et al. (2026)—searched the SAE latent space of Gemma-2-2B-IT and LLaMA-3.1-8B-IT for feature sets whose ablation flips refusal.

    Categorized refusal directions (closest to this paper)

    • Joad et al. (2026)—extracted directions for 11 refusal categories on Gemma-2-9B-IT and Llama-3.1-8B-Instruct, finding that the refusal efficacy steered by each direction is similar and that differences are mainly in refusal style (annotated with an LLM judge). The authors state that this paper fills in the self-referential category they did not study, and performs the dose-response, randomized controlled register transfer measurement they did not do.
    • Alagharu et al. (2026)—obtained category-aligned directions via refusal-token fine-tuning, used for multi-category refusal control.

    Causal analysis and steering methods

    • Vig et al. (2020)—introduced causal mediation analysis in NLP interpretability; activation steering methods follow Turner et al. (2024) and Rimsky et al. (2024) (contrastive activation addition), and the authors state that their direction injection is similar but different.
    • Tools: Gemma Scope SAE (Lieberum et al., 2024), TransformerLens (Nanda & Bloom, 2022); SAE methods based on Bricken et al. (2023) and Cunningham et al. (2024).

    Positioning: the authors state that this paper's distinction lies in adding the self-referential category, measuring register transfer with randomized-controlled dose-response curves, and characterizing the asymmetry between the two directions. The paper has no comparison baseline methods; controls are magnitude-matched random directions/random features.

  3. How does the paper address this problem?

    An SVM finds two category directions, then three types of intervention—SAE ablation, feature clamping, and DoM direction injection—test their causal role.

    Overall approach: on the residual stream at layer 20 of Gemma-2-9B-IT, use a linear SVM to obtain two category-boundary directions B1 (HARMFUL vs NORMAL) and B2 (SELFREF vs NORMAL), then test their causal roles with three types of intervention (ablation, steering, injection).

    Setup and direction extraction

    • Model and SAE: Gemma-2-9B-IT, with the Gemma Scope layer-20 SAE (width 16k, L0 ≈ 91).
    • Prompts: three categories, each with 10 fitting (FIT) + 10–20 evaluation (EVAL) prompts: HARMFUL (contraband synthesis, etc.), SELFREF (asking about the system prompt), NORMAL (common-knowledge questions).
    • Direction extraction: take the layer-20 activation of the last token of the assistant-turn prefix in the chat template, fit a linear SVM, and use the normal vector as the candidate direction. The authors explicitly state that the SVM is only a direction-discovery heuristic (n=10, 3584 dimensions, any linear classifier can separate them perfectly) and make no generalization claim.
    • SAE feature ranking: rank features by the activation difference si = āi(cat+) − āi(cat−), i.e., the difference in a feature's mean activation on the positive and negative classes.

    Three types of intervention

    • Ablation (necessity): hook at layer 20, encode through the SAE, set the top-K target features to zero, then decode.
    • Steering (sufficiency): clamp the top-K features to the target category's mean activation.
    • Direction injection: add α·s·v̂DoM to the last-token residual at layer 20, where v̂DoM is the unit-normalized difference-of-means direction for the corresponding category (the authors emphasize that what is injected is the DoM direction, not the SVM normal vector, and that "B1/B2 injection" is shorthand), and s is the natural projection magnitude; α ∈ {0.5, 1.0, 1.5, 2.0}, with a magnitude-matched random-direction control.

    Layer-wise analysis and stylistic measurement

    • Layer-wise ablation: repeat SAE ablation (K=250) at L ∈ {9, 20, 31}, comparing on-target/off-target effects of the two directions.
    • Stylistic measurement: two sets of regex word lists (no LLM judge): 22 patterns for content style and 17 patterns for meta style, with the metric being hits per 100 tokens.
    • Triple refusal metrics: refusal rate from an n-gram classifier (0–1), compliance from a gpt-4o-mini judge (0–5, and on a subset classified as REFUSAL/DEFLECTION/NEITHER), and regex word frequency; bootstrap 95% CI with 10,000 resamples.
  4. What experiments did the paper conduct?

    Ablation, steering, and injection experiments all show that the two directions are separable and asymmetric; B1 injection can rewrite the refusal register.

    Experimental setup

    • Models tested: Gemma-2-9B-IT (main experiments); Gemma-2-2B-IT (replication in Appendix J, 26 layers, d=2304).
    • Data: self-built HARMFUL / SELFREF / NORMAL three-category prompts, with 10 FIT and 10–20 EVAL per category; sufficiency experiments use n=50 NORMAL; injection sweeps use n=10 recipient prompts.
    • Judge: gpt-4o-mini structured output, giving compliance (0–5) and classifying refusal type on a subset; register uses a regex word list, with no LLM judge.
    • Controls: magnitude-matched random directions (3 seeds) and random feature ablation.
    • Statistics: bootstrap 95% CI (10,000 resamples); the paper does not report the number of repetitions (except 3 seeds for random controls).

    Main results

    Experiment (9B, layer 20, K=250)ResultSource
    cos(B1, B2)0.252 (about 75°)Section 3
    B1 ablation → HARMFUL refusal rate100% → 55% (SELFREF remains 100%)Figure 1
    B2 ablation → SELFREF refusal rate100% → 35% (HARMFUL remains 95%)Figure 1
    B1 ablation → HARMFUL compliance Δ+0.85, CI [+0.30, +1.50] (excludes 0)Table 1
    B2 ablation → SELFREF compliance Δ+0.20, CI [−0.15, +0.55]Table 1
    B1 injected into SELFREF, content word frequency (α=2.0)2.55 → 10.37/100 token; random control 1.13–2.16Figure 5
    B2 injected into HARMFUL, meta word frequency (α=2.0)0.21 → 2.75/100 token; random control 0.22Section 7
    • The authors state that the two directions are "substantially non-parallel" but not strictly orthogonal, and claim only "different rather than collinear."
    • The authors state that B1's causal effect is consistently larger than B2's across the three intervention types—ablation, sufficiency, and injection—and treat this asymmetry itself as a finding: B1 appears to encode both category detection and refusal vocabulary information, while B2 mainly encodes category detection.
    • After injection the model still refuses (refusal rate ≥ 0.80), outputs do not degenerate (degeneration = 0), and length is preserved (44–53 token); only the vocabulary changes.

    Sufficiency: steering benign prompts to produce refusal

    • What was measured: on NORMAL prompts, clamp the top-250 features to the target-class mean (n=50).
    • Results: B1 steering induces refusal in 8/50 (16%), B2 steering in 7/50 (14%), random feature control 0/150 (Figure 2); B1 induces content-style wording, B2 induces meta-style deflection (Figure 3).
    • Authors' interpretation: the magnitude is not large, but the n=50 sample and the clean random control indicate that the small effect is direction-specific.

    Layer-wise analysis: distinguishing two mechanisms

    • What was measured: repeat ablation at L ∈ {9, 20, 31} (K=250), testing "two independent circuits" (A) versus "two upstream classifiers feeding the same refuser" (B).
    • Results: at L9, B1's on-target effect is near zero (Δ=+0.05) and B2's on-target Δ=+0.35; at L20 this reverses (B1 +0.85, B2 +0.20); at L31, B1's on-target effect persists (+0.75) and leaks off-target to SELFREF (+0.65, CI [+0.05, +1.35]) (Figure 4).
    • Authors' interpretation: the results support (A); B2's effect is measurable from layer 9 and roughly flat, B1's emerges at layer 20 and persists to layer 31, and the late leakage is consistent with "layer 31 encoding refusal language production rather than content category."

    Refusal type shift and per-prompt distribution

    • Refusal type: at L20, K=250, B2 ablation changes SELFREF's REF/DEF/NEI from 65/35/0 to 40/50/10 (Table 2, n=20 HARMFUL + n=20 SELFREF); the authors state that ablation mostly turns formal refusals into deflection rather than full compliance.
    • Per-prompt distribution: at α=2.0, all 10 SELFREF prompts have content word frequency ≥ 3.8, median 11.3, IQR [9.7, 11.5] (Table 3); the authors state that the mean is not driven by outliers, and that the baseline mean of 2.55 is itself propped up by a single outlier of 13.6.

    Other ablations and analyses

    • Geometric control: directly fitting a HARMFUL-vs-SELFREF classifier B12 gives 5-fold CV accuracy 1.00; the NORMAL projection +2.9±17.2 lies between the two classes, ruling out a shared-anchor artifact (Section 3).
    • Relation to Arditi direction: cos(B1, rArditi)=0.68, cos(B2, rArditi)=0.24; 46% of B1's variance lies along rArditi (Section 3).
    • Injection depth: injection at L25/28/31 degenerates at lower α (100% degeneration at L31, α=1.5); only L20 remains coherent across all α (Section 7).
    • 2B replication: at L12, α=1.0, content word frequency 1.72→6.37 (random control 1.35), but the peak is lower than 9B (6.37 vs 10.37) and the coherent window is narrower (only L12, α∈[0.5,1.0]); B2→HARMFUL also shows no comparable effect in 2B (Table 5).
    • SVM vs DoM ranking: feature overlap is 50% at K=10 and drops to 16% at K=500 (Figure 6).
  5. What are some points that can be further explored?

    The authors list limitations such as a single family, small samples, and two categories, and propose four future directions including cross-family replication.

    Limitations and future directions noted by the authors

    • Single model family: only Gemma-2 is studied (2B replication in Section J) (Section 8, Limitations).
    • Small prompt sets: boundary fitting uses n=10 FIT and ablation evaluation uses 10–20 EVAL; the injection sweep uses n=10 receiver prompts (Section 8, Limitations).
    • Only two refusal categories are tested (Section 8, Limitations).
    • Stylistic measurement is regex-based: the content word list includes refusal phrases, so part of the stylistic effect is actually a shift toward formulaic content-style refusal wording; the authors state that using an LLM judge or human annotation for style could make the measurement independent of the word list (Section 8, Limitations).
    • Layer-wise analysis samples only 3 of 42 layers (L9, L20, L31) (Section 8, Limitations).
    • Future directions: replication across families (Llama, Mistral, Qwen); denser layer sweeps (e.g., every 3 layers from L9–L20); unsupervised refusal-type discovery via SAE feature clustering; controlled fine-tuning to test whether the asymmetry stems from training-data composition; injecting the SVM normal vector and the component of B1 orthogonal to rArditi are left for future work (Section 8).

    Scope of experiments

    • The models tested are Gemma-2-9B-IT and Gemma-2-2B-IT, 2 in total, both in the Gemma-2 family (Section 2, Appendix J).
    • Refusal categories cover HARMFUL and SELFREF, with NORMAL as control (Section 2).
    • Intervention layers cover L9/L20/L31 (ablation) and L20/L25/L28/L31 (injection) for 9B, and L12/L16/L18/L20 for 2B (Section 6, 7, Appendix J).
    • Sample sizes: 10–20 EVAL per ablation cell, sufficiency n=50, injection n=10 receiver prompts, random control 3 seeds; the paper does not report the number of repetitions for the main experiments.
    • LLM judging or human annotation of the stylistic effect was not done (the authors state the judge only rates compliance and refusal type); agreement between judges and humans is not reported in the paper.
  6. Summarize the main content of the paper

    In Gemma-2-9B-IT, the two class-aligned refusal directions are separable and asymmetric; B1 injection can rewrite refusal style while maintaining refusal.

    A mechanistic interpretability paper studying whether the refusal direction in instruction-tuned models encodes refusal style (how to refuse), not merely whether refusal occurs.

    • Question: Prior work established that a single direction (Arditi et al., 2024) and multiple/category directions (Wollschläger et al., 2025; Joad et al., 2026) mediate refusal; the authors additionally test the previously unstudied self-referential category, performing a dose-response stylistic transfer measurement with a randomized control.
    • Method: At layer 20 of Gemma-2-9B-IT, use an SVM to extract B1 (harmful vs neutral) and B2 (self-referential vs neutral), then test causal effects via three types of intervention—SAE feature ablation, feature clamping steering, and DoM direction injection—with metrics being refusal rate, gpt-4o-mini compliance, and regex word frequency.
    • Geometry: cos(B1, B2)=0.252 (about 75°), with cosines to the Arditi direction of 0.68/0.24 respectively; the authors state that the Arditi single direction is better understood as a mixture of multiple category-aligned components.
    • Necessity: K=250 ablation selectively reduces the refusal rate for this category (B1→HARMFUL 55%, B2→SELFREF 35%), and the two directions have different layer profiles (Figure 1, 4).
    • Stylistic injection (main result): Injecting the B1 direction into SELFREF prompts, content word frequency rises monotonically from 2.55 to 10.37/100 tokens (α=2.0), random control 1.13–2.16; the model still refuses but switches to content-style wording; the reverse effect of B2 is about an order of magnitude smaller, and the authors state the asymmetry is direction-driven rather than magnitude-driven.
    • Replication and conclusion: 2B qualitatively replicates at L12 (1.72→6.37, narrower window). The authors state that refusal is not monolithic and that style can be causally controlled, and recommend that jailbreak robustness benchmarks report metrics by refusal category; limitations include a single model family, small samples, two categories, and a regex-based stylistic measurement.
Read originalarxiv.org