Skip to content

Safety side effects of preventive personality steering: an empirical audit

Safety Side Effects of Preventative Persona Steering: An Empirical Audit

arXiv: 其他 AI 安全论文· arXiv:2611.17620· Sabrina Tang, Nikolai Vo, Carson Murillo, Chira Sukumarnpiti, Sarah WilsonUniversity of California, Irvine; University of Toledo; University of California, San Diego; Berkeley City College; GoogleIndexed by this site · 4 days ago· Originally publishedAlignment· Its place in the scan record →

Quick read

Analysis and theory

Based on the paper PDF (AI-generated); refer to the original

The authors audit the safety side effects of preventive persona steering, stating that it still changes refusal behavior when MMLU and coherence are unchanged, with the magnitude depending on the trait.

Questions
Preventative persona steering is recommended because it adds the vector during fine-tuning and is claimed to preserve MMLU, but existing validation measured only capability, not refusal safety; the authors ask whether it changes refusal robustness when capability, coherence, and trait suppression are all normal.
Methods
Using the original pipeline of Chen et al. [2026] to extract sycophancy and hallucination vectors, compare three conditions—fine-tuned base, preventative steering, and inference-time steering—on three models, sweeping coefficient c from 0 to 3, while measuring HarmBench, StrongREJECT, XSTest, trait expression, coherence, and MMLU.
Experiments and results
The authors report: hallucination steering reduced HarmBench ASR for three models from 61.0–91.2% to 4.4–22.0%, far greater than the reduction in the trait; sycophancy steering caused only small changes and the directions were inconsistent across benchmarks; MMLU changed by no more than 2.1 percentage points.
Limitations and future work
The authors note: each point is a single run, the sweep is relatively coarse, and no confidence intervals are given; the 14B layers were scaled proportionally from 7B rather than searched; the conclusions may not generalize to other benchmarks, adaptive attacks, or model families (Section 4).
Experimental setupQwen2.5-7B-Instruct、Llama-3.1-8B-Instruct et al. · HarmBench、StrongREJECT et al. · HarmBench ASR、StrongREJECT score et al.
Evaluated models
Qwen2.5-7B-InstructLlama-3.1-8B-InstructQwen2.5-14B-Instruct
Benchmarks
HarmBenchStrongREJECTXSTestMMLU
Metrics
HarmBench ASRStrongREJECT scoreXSTest unsafe compliancetrait expressioncoherenceMMLU
AI summaryThe paper empirically audits the safety side effects of preventive personality steering, finding that refusal behavior still changes on three models, with the magnitude depending on the trait being targeted. Personality vectors are activation-space directions corresponding to high-level behaviors such as sycophancy and hallucination; preventive steering adds the vector during fine-tuning so that the optimizer does not need to write the trait into the weights, and then removes it at evaluation time. Chen et al. (2025) recommended this method because it preserves MMLU accuracy, but existing validation only measured capability. The authors perform preventive steering on sycophancy and hallucination vectors while measuring three safety benchmarks as well as trait expression, coherence, and MMLU, concluding that preventive personality steering cannot be proven to be safely neutral and recommending that steering methods be evaluated with multiple safety benchmarks.

The paper empirically audits the safety side effects of preventive personality steering, finding that refusal behavior still changes on three models, with the magnitude depending on the trait being targeted. Personality vectors are activation-space directions corresponding to high-level behaviors such as sycophancy and hallucination; preventive steering adds the vector during fine-tuning so that the optimizer does not need to write the trait into the weights, and then removes it at evaluation time. Chen et al. (2025) recommended this method because it preserves MMLU accuracy, but existing validation only measured capability. The authors perform preventive steering on sycophancy and hallucination vectors while measuring three safety benchmarks as well as trait expression, coherence, and MMLU, concluding that preventive personality steering cannot be proven to be safely neutral and recommending that steering methods be evaluated with multiple safety benchmarks.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Audit whether preventive persona steering still changes refusal safety when capability metrics are normal.

    Core question: whether preventative persona steering changes the model's refusal behavior when MMLU, coherence, and trait suppression all appear successful.

    • Background: persona vectors are directions in activation space corresponding to traits such as sycophancy and hallucination [Chen et al., 2026], which can be subtracted at inference time (inference-time steering) or added during fine-tuning and removed at evaluation (preventative steering).
    • Gap in existing validation: Chen and Arditi [2025] recommend preventative steering because it preserves MMLU, but the authors note that this recommendation validated only capability and did not evaluate refusal safety; meanwhile, activation steering is known to have unexpected safety side effects [Xiong et al., 2026; Korznikov et al., 2025; Li et al., 2026].
    • Research questions: RQ1 asks whether preventative steering changes refusal robustness; RQ2 asks whether the safety benchmarks agree on the direction of change; RQ3 asks whether the effect is consistent across traits; RQ4 asks whether it is consistent across model families and scales.
    • What the paper does: on Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct (the two models used by Chen and Arditi [2025]), and Qwen2.5-14B-Instruct (to test scale), audit preventative steering of sycophancy and hallucination vectors, measuring three safety benchmarks, trait expression, coherence, and MMLU simultaneously.
    • Authors' core conclusion: preventative steering still changes refusal behavior, with magnitude depending on the trait; capability preservation does not show that it is safety-neutral.
  2. What related research exists?

    Builds on two lines—persona vectors and preventative steering—as well as research on safety side effects of steering and on layer-selection reliability.

    Persona vectors and preventative steering

    • Chen et al. [2026]—propose persona vectors, using activation-space directions to monitor and control language model traits; this paper uses their unmodified pipeline to extract sycophancy and hallucination vectors, and follows their trait-expression and coherence scoring pipeline.
    • Chen and Arditi [2025]—follow-up experiments on preventative steering, recommending the method because of capability preservation; the authors note that they did not evaluate refusal safety, and this paper fills exactly this gap, using the same two models as theirs.

    Safety side effects of steering

    • Xiong et al. [2026], Korznikov et al. [2025], Li et al. [2026]—find that steering directions chosen to modify a certain behavior change refusal or weaken safety guardrails, even when the intervention succeeds on the target behavior. The authors note that these studies perform inference-time steering without persona vectors, and the interaction between preventative persona steering and refusal robustness has not been directly studied.

    Layer selection and reliability

    • Heyman and Vandeputte [2026]—find that steering effects vary greatly across layers and token positions; this paper adopts their layer selection (layer numbering starts from 1, following their Appendix C).
    • Tan et al. [2024]—point out that steering's sensitivity to coefficients and prompts is a key reliability issue; this paper sweeps coefficients as they recommend, but defines reliability as side effects on refusal rather than the success rate on the target behavior.

    Evaluation tools (used as measurement instruments)

    • HarmBench [Mazeika et al., 2024] (ASR, labeled by the official classifier HarmBench-Llama-2-13b-cls), StrongREJECT [Souly et al., 2024] (GPT-4o-mini scores according to its rubric), XSTest [Röttger et al., 2024] (GPT-4 labels full compliance/partial refusal/full refusal).

    The authors position this paper as: extending steering reliability research from inference time to fine-tuning time, and changing the reliability criterion from success rate on the target behavior to refusal side effects on three safety benchmarks.

  3. How does the paper address this problem?

    Three conditions × two traits × three models, sweeping coefficient c∈{0,…,3}, measuring both safety benchmarks and capability metrics.

    Overall approach: keep the vector extraction pipeline fixed, change only how the vector is applied, and at each point of the coefficient sweep simultaneously measure safety benchmarks, trait expression, coherence, and MMLU, to see whether safety changes are masked by capability and trait metrics.

    Vector extraction

    • Pipeline: use the unmodified pipeline of Chen et al. [2026] to extract sycophancy and hallucination vectors; the authors state that this way safety differences reflect only how the vector is applied rather than how it is extracted.

    Three conditions

    • Fine-tuned base: fine-tune on trait data without steering, used to separate the effect of steering from the effect of fine-tuning itself.
    • Preventative steering: during the same fine-tuning process, add the vector to the residual stream via a forward hook; at c = 0 it is equivalent to the fine-tuned base.
    • Inference-time steering: subtract the vector from the residual stream during generation with the original Instruct model, without fine-tuning; because the vector is extracted from the original model's activations, the authors state that this separates the vector's causal effect from changes caused by fine-tuning.
    • Coefficient: the two steered conditions sweep c ∈ {0, 0.25, …, 3}.

    Layer selection

    • 7B and 8B: follow Chen et al. [2026] and Heyman and Vandeputte [2026]—Qwen2.5-7B-Instruct uses layer 20 (sycophancy) and layer 16 (hallucination); Llama-3.1-8B-Instruct uses layer 16 for both traits; layer numbering starts from 1.
    • 14B: neither source covers Qwen2.5-14B-Instruct; the authors scale the 7B layers from 28 layers to 48 layers by relative depth, obtaining layer 34 (sycophancy) and layer 27 (hallucination).
    • For each model and trait, only one residual stream layer is steered, and both steering methods use the same layer.

    Evaluation and judging

    • Preventative steering: for each coefficient, measure HarmBench ASR, StrongREJECT score, compliance rate on XSTest unsafe prompts, trait expression, coherence (persona-vector scoring pipeline), and MMLU.
    • Inference-time steering: only trait expression and coherence are measured.
    • Judging method: HarmBench is labeled by the official classifier HarmBench-Llama-2-13b-cls; StrongREJECT is scored by GPT-4o-mini according to its rubric; XSTest is labeled by GPT-4 as full compliance, partial refusal, or full refusal; the authors manually spot-check judge labels to distinguish harmful compliance from noise or incoherent outputs.
  4. What experiments did the paper conduct?

    Hallucination steering greatly reduces ASR while trait expression drops only slightly; sycophancy steering changes little and the direction is inconsistent across benchmarks.

    Experimental Setup

    • Models tested: Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Qwen2.5-14B-Instruct, 3 in total.
    • Traits and conditions: two persona vectors, sycophancy (Syc.) and hallucination (Hal.); three conditions: fine-tuned base, preventive steering, and inference-time steering; the steered conditions sweep c ∈ {0, 0.25, …, 3}.
    • Benchmarks and metrics: HarmBench ASR, StrongREJECT score, XSTest unsafe prompt compliance rate, trait expression and coherence (%), MMLU; the paper does not report the number of samples for each benchmark.
    • Evaluation method: HarmBench official classifier HarmBench-Llama-2-13b-cls; StrongREJECT scored by GPT-4o-mini according to a rubric; XSTest three-way classification by GPT-4; the authors manually spot-checked judge labels.
    • Number of repetitions: the paper explicitly states that each point is a single run, and no confidence intervals are given.

    Main Results

    Changes in HarmBench ASR for preventive steering from c = 0 to c = 3 (main text Section 3; c = 0 is the fine-tuned base):

    Swipe horizontally to read the table

    ModelTraitc=0c=3Notes
    Qwen2.5-7B-InstructHallucination72.3%4.4%StrongREJECT decreases in the same direction
    Llama-3.1-8B-InstructHallucination61.0%22.0%Non-monotonic: lowest 14.5% at c=2.25
    Qwen2.5-14B-InstructHallucination91.2%6.3%StrongREJECT decreases in the same direction
    Llama-3.1-8B-InstructSycophancy8.2%11.9%Relatively up 46%; XSTest moves in the opposite direction from 8.5% to about 6.5%
    Qwen2.5-7B-InstructSycophancy20%14%StrongREJECT 17%→5%; XSTest slightly up
    Qwen2.5-14B-InstructSycophancy48%48%StrongREJECT 67%→61%; XSTest 42%→59%
    • Authors' interpretation (hallucination): the hallucination vector has a stronger effect on refusal behavior than on the hallucination trait itself—trait expression remains ≥70% over the same interval, and most of the safety change occurs at c ≤ 1.25, where trait expression is still above 96%; the authors say this coupling is invisible to capability and trait metrics.
    • Authors' interpretation (sycophancy): for each model, at least one benchmark moves in the opposite direction from another benchmark, and no single benchmark can characterize the effect.
    • Capability metrics: Figure 1 shows that coherence is flat or slightly up across sweeps, and MMLU changes by at most 2.1 percentage points (Qwen2.5-7B 69.6–70.3%, Llama-3.1-8B 58.0–62.9%, Qwen2.5-14B 74.7–76.2%); the authors say none of the above safety changes are accompanied by capability degradation.

    Divergence Between XSTest and Attack-Type Benchmarks

    • What was measured: under preventive hallucination steering, the comparison of XSTest unsafe compliance rate with HarmBench and StrongREJECT.
    • Results: HarmBench and StrongREJECT approach zero on the two Qwen models, but the XSTest unsafe compliance rate stays at about 25–30% on Qwen2.5-7B-Instruct and at 37–43% on Qwen2.5-14B-Instruct.
    • Authors' interpretation: steering restores refusals for the attack types covered by the attack-type benchmarks, but does not restore refusals for all unsafe requests.

    Trait Expression and Coherence (Table 1)

    • Fine-tuning raises the trait: fine-tuning on trait data raises trait expression from 4–34% for the original Instruct models to 89–100% (c = 0 column).
    • Preventive steering partially suppresses it: at c = 3, trait expression drops to 27.9–94.6% (lowest is Llama sycophancy 27.9%, highest is Llama hallucination 94.6%), and coherence is flat or slightly up.
    • Inference-time steering suppresses more strongly: at c = 3, trait expression is all pushed below 6%, and coherence is above 89% except for Llama hallucination; Llama hallucination coherence drops to 39.7% at c = 3. The authors note that inference-time steering acts on the original model, whose trait interval does not overlap with that of the fine-tuned model, so the two methods cannot be calibrated to the same trait level.

    Other Ablations and Analyses

    • Fine-tuning baseline effect: at c = 0, HarmBench ASR on the same base varies with the trait data—Llama 8% vs. 61%, Qwen-7B 20% vs. 72%, Qwen-14B 48% vs. 91% (sycophancy vs. hallucination data).
    • Authors' interpretation of the baseline effect: fine-tuning on hallucination data itself appears to erode refusals, and the effect of preventive steering on this trait may partly offset this erosion.
    • Non-monotonic exception: for Llama hallucination steering, ASR reaches its lowest 14.5% at c = 2.25 and rises back to 22.0% at c = 3; the paper gives no reason.
  5. What are some points that can be further explored?

    The authors list limitations such as single runs, coarse sweeps, and 14B layer scaling; the experiments cover three models, two traits, and three benchmarks.

    Limitations and future directions noted by the authors

    • Single runs and coarse sweep: each point is one run, the sweep is relatively coarse, and no confidence intervals are given, limiting claims about the shape of the dose-response (Section 4).
    • 14B layer selection: the layers for Qwen2.5-14B-Instruct were selected by scaling the 7B choices proportionally rather than by sweeping, so its results may partly reflect suboptimal layers (Section 4).
    • Benchmark coverage: HarmBench, StrongREJECT, and XSTest probe different aspects of safety, and the conclusions may not generalize to other benchmarks, adaptive attacks, or model families (Section 4).
    • Evaluation recommendation: the authors believe that evaluating persona vectors should include multiple safety benchmarks in addition to capability and target-trait metrics (Section 4, Section 5).

    Scope of the experiments

    • The models tested are Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Qwen2.5-14B-Instruct, 3 in total; the traits are sycophancy and hallucination, 2 kinds (Section 1, 2).
    • Safety evaluation covers three benchmarks: HarmBench, StrongREJECT, and XSTest; the inference-time steering condition measured only trait expression and coherence, not safety benchmarks (Section 2).
    • The coefficient sweep is c ∈ {0, 0.25, …, 3}, and for each model and trait only one residual stream layer is steered (Section 2).
    • Judging consists of three automatic judges—HarmBench-Llama-2-13b-cls, GPT-4o-mini, and GPT-4—plus manual spot-checking; the paper does not report the sample size or agreement numbers of the manual spot-check (Section 2).
    • The paper does not report the number of samples per benchmark, nor the fine-tuning hyperparameters or data scale.
  6. Summarize the main content of the paper

    The audit found that preventive persona steering still changes refusals when capability metrics are normal, and the direction and magnitude depend on the trait and the benchmark.

    An empirical audit of the safety side effects of activation steering: preventative persona steering is recommended for preserving MMLU, but its refusal-safety consequences had not been directly studied before.

    • Approach: use the original pipeline of Chen et al. [2026] to extract sycophancy and hallucination vectors, compare fine-tuned base, preventative steering, and inference-time steering on Qwen2.5-7B/14B-Instruct and Llama-3.1-8B-Instruct, sweeping coefficient c from 0 to 3, while measuring HarmBench, StrongREJECT, XSTest, trait expression, coherence, and MMLU.
    • Hallucination steering: HarmBench ASR on the three models drops from 61.0–91.2% to 4.4–22.0%, while trait expression remains ≥70%; most of the change occurs at c ≤ 1.25, where the trait remains above 96%. The authors state that the vector's effect on refusal is stronger than on hallucination itself.
    • Sycophancy steering: safety metrics move only slightly and the direction varies by model and benchmark, e.g., on Llama, HarmBench ASR rises from 8.2% to 11.9% while XSTest compliance rate drops from 8.5% to about 6.5%; for each model, at least one benchmark moves in the opposite direction from another benchmark.
    • Capability metrics: across all sweeps, MMLU changes by no more than 2.1 percentage points, and coherence stays flat or rises slightly, i.e., safety changes are not accompanied by capability degradation.
    • Exceptions: Llama hallucination steering is non-monotonic (ASR is lowest at 14.5% at c = 2.25); XSTest on the two Qwen models stays at about 25–43% and does not approach zero along with the attack-type benchmarks.
    • Authors' conclusion: capability preservation does not show that preventative persona steering is safety-neutral, and steering methods should be evaluated on multiple safety benchmarks; limitations include single runs, coarse sweeps, and 14B layers scaled proportionally, and the conclusions may not generalize to other benchmarks, adaptive attacks, or model families.
Read originalarxiv.org