Analyzing the safety pitfalls of steering vectors

Does activation steering change the safety alignment of LLMs, and where does the change come from.

• Scenario: The authors state that steering modifies activations at inference time without updating weights, and has been used for truthfulness, sycophancy, political orientation, and complex reasoning. Brittleness has been documented, but safety consequences have rarely been systematically examined.
• Existing gaps: Existing analyses mostly start from utility, discussing high variance, poor out-of-distribution generalization, effectiveness only on specific tasks, and whether it is more useful than prompting. The authors argue this misses a more dangerous point: weak prompt-level jailbreaks become more likely to succeed due to steering.
• Core observation: Steering toward a specific behavior direction can increase or decrease ASR. The authors attribute this to the overlap between the steering vector and the latent direction of refusal behavior, rather than the semantics of the behavior name alone.
• Proposed audit: Using Contrastive Activation Addition (CAA) as the target, audit open-source models from 3 families and 6 scales (3B–32B) with a unified protocol; use template attacks as diagnostics, provide a geometric interpretation and ablations, and discuss the trade-off between controllability and safety.
• Audit setup:
  • The intervention is on the residual stream, vectors come from behavior contrasts, and strength is controlled by a scalar multiplier.
  • Probes are harmful requests from JailbreakBench, as well as Refusal Suppression and Prefix Injection; there is also prompt-only without attacks.
  • Responses are classified by GPT-4o-mini as safe, jailbroken, or non-comprehensible, then ASR is computed; FRR is computed on benign requests.
  • The paper does not use terms such as white-box or black-box.

Construction of steering

• Activation difference methods: Turner et al. (2024), Panickssery et al. (2024), and Zou et al. (2025) modify behavior using activation differences without changing parameters; Subramani et al. (2022) extract latent steering vectors.
• Other intervention levels: Li et al. (2023) and Zhang et al. (2024) modify attention head outputs; Chalnev et al. (2024) use sparse autoencoders to extract residual stream features. The paper only describes these, without comparing their safety metrics to CAA.
• Existing applications: Mitigating sycophancy (Panickssery et al., 2024), political bias (Kim et al., 2025), complex reasoning (Wang et al., 2025). The introduction uses these to illustrate the uses of steering.

How fragility is discussed

• Variability and generalization: Tan et al. (2024) discuss high variability and poor out-of-distribution generalization; Silva et al. (2025) discuss frequent ineffectiveness.
• Task scope: Brumley et al. (2024) state that effects are often limited to specific task types; van der Weij et al. (2024) state that steering is less successful when shifting multiple behaviors simultaneously.
• Geometry and prompting: Braun et al. (2025) link causes to the geometric consistency of activation differences; Wu et al. (2025) compare prompting and question the utility of steering relative to simple baselines. The paper notes that these works primarily treat fragility as a utility problem.

Alignment, jailbreaks, and refusal representations

• Refusal training: Ouyang et al. (2022), Bai et al. (2022a,b), Rafailov et al. (2023). Barnhart et al. (2025), Ji et al. (2024), and Wolf et al. (2024) discuss alignment fragility; Qi et al. (2024), Su et al. (2024), and Wei et al. (2024) argue that unsafe capabilities are suppressed rather than erased.
• Prompt-level jailbreaks: Wei et al. (2023), Huang et al. (2023), Andriushchenko et al. (2025), Zou et al. (2023). The two template attacks come from Wei et al. (2023).
• Refusal direction: Arditi et al. (2024) state that refusal is mediated by a single direction in the residual stream, and this construction is adopted. Wollschläger et al. (2025) and Pan et al. (2025) argue that refusal is higher-dimensional. Gao et al. (2025), Kawasaki et al. (2025), and Li et al. (2025) analyze how refusal is subverted in activation space.

Experimental control: The control is the unsteered model with m=0, not another steering algorithm. The benchmark is JailbreakBench.

The authors state that this work extends the representation perspective to systematic shifts in ASR due to steering, and uses partial alignment with the one-dimensional refusal direction to examine how activation shifts leave the safety manifold.

CAA behavior vectors are added to the residual stream; ASR changes are then explained via overlap with the refusal direction, and an ablation that modifies only the vector tests whether that component is involved in the safety change.

Intervention form

• The target is the residual stream of a decoder-only transformer. The activation at layer ℓ and position i is denoted h, and a fixed direction is added at inference time.
• The intervention is h̃⁽ⁱ⁾_ℓ(x)=h⁽ⁱ⁾_ℓ(x)+m v_ℓ,τ. m determines sign and strength, and v corresponds to behavior τ.
• Section 3 states the addition is applied to prompt tokens; the experimental protocol in Section 4.2 states it is added to all token positions after the prompt. The paper does not explain this discrepancy.
• In the main experiments, m takes values 0, ±0.5, ±1.0, ±1.5; m=0 means no steering.

Behavior vectors and layers

• The formalization follows CAA from Panickssery et al. (2024): only the answer choices are changed for the same question, residual activations at the answer letter are taken, and positive examples minus negative examples are averaged.
• The experimental vectors come from publicly released behavior vectors (Panickssery et al., 2024; Tan et al., 2024), 9 in total: Hallucination, Myopic Reward, Corrigible, Survival Instinct, Sycophancy, Self Awareness, Anti-LGBTQ, Coordinate-AI, Openness. Original identifiers are in Appendix A.3.
• The authors state these dimensions are relevant to alignment discussions, but are not explicit refusal signals, to avoid trivial overlap with the refusal direction.
• Layer selection: layers are swept with m∈{-1,0,+1} on held-out benign prompts, retaining layers with strong controllability; the ablation then targets the layer of the most prominent refusal direction. Appendix Table 3: Llama-7B 14/32, Gemma-7B 14/28, Qwen-3B 27/36, Qwen-7B 19/28, Qwen-14B 30/48, Qwen-32B 46/64.

Refusal direction

• Following Arditi et al. (2024): the mean difference in activations between harmful and harmless prompts at the last token gives per-layer candidates r_ℓ, and one r̂ is selected.
• Appendix A.4: harmful set has 128 training and 32 validation examples, from AdvBench, MALICIOUSINSTRUCT, TDC2023, HarmBench; harmless set likewise 128/32, from Alpaca. After filtering, there is no overlap with the evaluation benchmarks.
• On the validation set, three scores are used: bypass, induce, KL. The selection requires the lowest bypass, with induce_score>0, kl_score<0.1, and layer index<0.8L. The authors treat this as a traceable proxy for the refusal direction.

Ablation and evaluation

• The ablation acts only on the steering vector, not on the entire residual stream at inference time: v^⊥_ℓ,τ=v_ℓ,τ−(v_ℓ,τ^⊤r̂_ℓ)r̂_ℓ. The orthogonal component is retained.
• ASR is the proportion of jailbroken responses among comprehensible outputs to harmful requests; FRR is the proportion of benign requests incorrectly refused. The judge model is GPT-4o-mini, with the template in Appendix A.6.
• Attack templates are in Appendix A.5: Refusal Suppression forbids apologies, disclaimers, and refusal language; Prefix Injection requires restating a given opening. The template text is not reproduced here.

Experimental Setup

• Models: Llama-2-7b-chat-hf, Gemma-7b-it, Qwen2.5-7B-Instruct, and the 3B, 14B, 32B variants of the Qwen2.5 family. The full model IDs for the latter three are not given in the main text.
• Data: JailbreakBench, described in the paper as 100 harmful and benign examples, covering harassment, misinformation, privacy, etc. The number of items per cell is not separately specified.
• Attacks and Behaviors: prompt-only, Prefix Injection, Refusal Suppression; 9 behavior vectors. The control is m=0.
• Metrics: ASR, FRR, and ∆ASR=ASR(1.5)−ASR(0). Also reported are cosine similarity with r̂, and OLS on the ASR slope. The slope follows Equation 4, taking the difference in ASR at m = -1.5 and 1.5, divided by the difference in the multipliers at the two ends.
• Evaluation: GPT-4o-mini three-class classification. Appendix A.6 states that an affirmative sentence alone is insufficient to determine jailbroken. In 99.8% of evaluation files, incomprehensible outputs are below 5%, and 100% below 20%.
• Execution: Appendix A.1 specifies single greedy decoding on a single A100 80GB. The paper does not report agreement between evaluation and human raters.

Main Results

Figure 3 shows ∆ASR for m=1.5 relative to m=0. The table below only includes the four prefix injection columns; column order follows the bottom labels of the figure.

Swipe horizontally to read the table

Model	Sycophancy	Self-Awareness	Openness	Anti-LGBTQ
Qwen 3B	0%	0%	0%	0%
Gemma 7B	18%	28%	10%	-3%
Llama 7B	15%	6%	6%	-1%
Qwen 7B	28%	28%	29%	-24%
Qwen 14B	56%	48%	57%	-15%
Qwen 32B	49%	47%	54%	-4%

• The authors state that template attacks amplify steering, and larger models in the Qwen family show larger changes. In the table, Qwen 3B has 0% in these four cells; Qwen 14B's Openness is 57%, the largest increase in Figure 3.
• In the same figure under refusal suppression, Anti-LGBTQ for Qwen 14B is -50%, the largest decrease. The authors state that Sycophancy and Openness mostly increase ASR, while Coordinate-AI and Anti-LGBTQ mostly decrease ASR, and semantic and safety effects do not correspond one-to-one.
• The authors state that prompt-only typically has |∆ASR|<15%. In Figure 3, Qwen 14B's Sycophancy is 38% and Qwen 7B's is 17%, which are outside this range. Figure 1 marks Qwen 14B's ASR without steering as 4%, with only Self-Awareness +1.5 at 42%, and with the ADV prefix at 80%; the paper does not specify which column of Figure 3 these three cells correspond to in absolute ASR.

Correlation with Refusal Direction

• Cosine Similarity: In Figure 4, Sycophancy and Self-Awareness, which increase ASR, are negatively aligned with r̂, while Anti-LGBTQ, which decreases ASR, is positively aligned. The authors interpret this as a geometric overlap mechanism, not behavioral semantics.
• Prefix Injection: In Table 1, Qwen 3B has a slope of 1.46, r=0.15, R²=0.02, p=0.696. The other five models have p<0.001; R² ranges from 0.85 for Qwen 7B to 0.91 for Qwen 14B. Slopes range from -36.52 for Llama 7B to -193.40 for Qwen 32B. The authors state that larger models are more sensitive to this alignment.
• Other Two Settings: In Table 6, prompt-only shows negative correlation for all 6 models, with R² from 0.61 for Qwen 14B to 0.92 for Qwen 7B, and Qwen 3B also has p<0.01. In Table 7, refusal suppression shows Qwen 14B with R²=0.98, r=-0.99; Gemma 7B has R²=0.64, p<0.01, lower than the other models in the same table.

Ablations and Benign False Rejections

• Ablation Decreases: In Table 2, the average |∆ASR| decrease across the three attacks is Qwen 3B 10.86%, Llama 7B 24.63%, Gemma 7B 14.70%, Qwen 7B 19.81%, Qwen 14B 19.76%, Qwen 32B 19.09%, all in the downward direction. In the table, Mean |∆ASR| is without a percent sign, e.g., Qwen 14B's prefix injection goes from 20.64 to 17.43.
• Exceptions and Non-Recovery: The authors state that for sizes above 3B, decreases are mostly 15%–25%. Gemma's refusal suppression is 11.32%, and Qwen 3B's prefix injection is 2.21% (from 1.43 to 1.40). The authors state that ablation does not fully return to the ASR without steering, and argue that the one-dimensional r̂ does not cover all relevant safety subspaces.
• FRR: Figure 6 only plots Qwen models. Self Awareness slope=-2.893, Anti-LGBTQ slope=3.589. The authors state that when negatively aligned with the refusal direction, increasing the multiplier decreases FRR; when positively aligned, it pushes toward over-refusal. The figure does not label the FRR values at each point.

Other Ablations and Analyses

• Negative m=-1.5 (Figure 8): The authors state the direction roughly reverses; the magnitudes do not oppose Figure 3 cell by cell. Anti-LGBTQ, refusal suppression, Qwen 14B is 22%, while the positive same cell is -50%.
• Table 5 only reports Myopic Reward, Anti-LGBTQ, Self Awareness at ±1.5 for MMLU and TriviaQA. Each Qwen's TriviaQA Max |∆| is 0.0; MMLU's Max |∆| is Llama 7B 8.5, Qwen 7B 12.3. The authors state these are usually below 5%, but these two cells are not.
• Figure 9: The authors state that after ablation, the ASR curves for Anti-LGBTQ and Self-Awareness on Qwen are smoother near m=0; the figure does not label endpoint ASR values.
• Appendix C: Qwen-14B, sycophancy, m=+1.5, greedy decoding. prompt-only still refuses; both template attacks with steering produce harmful outputs. The appendix gives examples, not repeated here.

Limitations and future directions noted by the authors

• The refusal direction uses a one-dimensional vector as a proxy, which may not capture the full complexity of the safety mechanism; after ablation, perturbations remain, and the authors suggest the refusal subspace may be multidimensional. Future work could extend to higher-dimensional refusal subspaces (Section 10 Limitations).
• Section 8 offers two explanations: r̂ may still overlap with additional safety dimensions; even if the injected vector is orthogonal to the refusal direction at layer ℓ, later attention and MLP layers may reintroduce refusal components.
• The experiments only audit CAA. The authors argue that other activation interventions also modify the residual stream, so the results may represent a broader class; they still recommend applying the same audit to attention head interventions and sparse autoencoder steering (Section 10).
• The authors call for geometrically more robust alignment and steering methods that account for the model's safety geometry (Section 8 Discussion, Section 9 Conclusion).

Experimental coverage

• The tested models are Llama-2-7b-chat-hf, Gemma-7b-it, Qwen2.5-7B-Instruct, and Qwen2.5 3B, 14B, 32B, totaling 6 models (Section 4.1).
• Main results use JailbreakBench; attacks are prompt-only, Prefix Injection, and Refusal Suppression. There are 9 behavior vectors, with multipliers of 0 and ±0.5, ±1.0, ±1.5 (Sections 4.1–4.2).
• For each model, the main analysis uses one layer from Table 3. The refusal direction is split into 128/32 per Appendix A.4, constructed from AdvBench and Alpaca.
• Evaluation uses GPT-4o-mini; Appendix A.1 specifies single greedy decoding. The paper does not report agreement between the evaluator and humans.
• Capability numbers come from MMLU and TriviaQA (Wikipedia split); Table 5 includes only 3 behaviors. Ablation removes the refusal component from the steering vector, not ablating the residual stream at inference time (Section 7).

This is a safety audit of CAA steering: it changes jailbreak success rates, and the authors link the changes to geometric overlap with the refusal direction.

• Problem: Activation steering controls behavior at inference time without changing weights. The authors investigate whether this erodes safety alignment, not just whether utility is stable.
• Method: Using 9 publicly available CAA behavior vectors on 6 open-source models, they measure ASR and FRR with JailbreakBench and two template attacks. They then extract a one-dimensional refusal direction, regress ASR slopes against cosine similarity, and ablate that component from the steering vectors.
• Magnitude: In Figure 3, at m=1.5, the highest ∆ASR is 57% for Qwen 14B, Openness, prefix injection, and the lowest is -50% for the same model, Anti-LGBTQ, refusal suppression. Figure 1 shows that for the same model, Self-Awareness +1.5 with ADV prefix yields an ASR of 80%, with steering alone 42%, and without steering 4%.
• Mechanism: Under prefix injection, except for Qwen 3B (p=0.696, R²=0.02), ASR slopes are negatively correlated with cosine similarity to the refusal direction, with R² ranging from 0.85 to 0.91 (Table 1). After ablation, the average |∆ASR| reduction across the three attacks is highest for Llama 7B at 24.63% (Table 2), but baseline safety is not fully restored.
• Authors' conclusion: Steering is not safety-neutral; there is a trade-off between controllability and safety. FRR on benign requests also changes with the same geometric relationship. The authors call for steering that accounts for safety geometry and for geometrically more robust alignment.