WRDO: Weighted Refusal Direction Optimization in Aligned Large Language Models

1. 01What problem does this paper attempt to solve?

   The authors study whether combining per-prompt refusal directions with learned aggregation weights can yield a stronger refusal intervention than a single difference-of-means direction.

   Core question: standard single-direction refusal ablation averages heterogeneous harmful prompts into one direction; the authors study whether learning aggregation weights over per-prompt directions can construct a stronger refusal intervention.

   • Problem setting: prior work holds that the refusal behavior of aligned LLMs is mainly mediated by a single direction in the residual stream, that ablating one difference-of-means vector can disable refusal across multiple open-source models, and that this intervention can be written into the weights once with no inference-time overhead (Introduction).
   • Shortcoming of the existing construction: the authors argue that the standard construction puts harmful prompts of different categories—misinformation, malware, privacy violation, etc.—into the same mean, so the resulting refusal direction is "diffuse" and conflates phenomena that may not be aligned in the residual stream (Section 1).
   • Core hypothesis: per-prompt directions are qualitatively heterogeneous, and the authors cite experience from ensemble classifiers, federated averaging, and adapter selection to argue that performance-aware weighted aggregation outperforms uniform averaging when component quality is uneven, so learning aggregation weights may yield a stronger intervention direction (Section 1).
   • What the paper proposes: Weighted Refusal Direction Optimization (WRDO), which learns aggregation weights from a library of per-prompt directions to construct a composite refusal direction, with two instances, WRD-CMAES and WRD-GRPO-IS; it also studies a defense setting that applies the same mechanism in reverse to restore refusal behavior in abliterated models (Section 1).
   • Threat model (as stated in the paper):
     • Attacker goal: suppress the refusal behavior of aligned models via weight-space intervention so that they give unsafe answers to harmful prompts (measured by ASR, Section 2.4).
     • Knowledge and capability: operate on the weights of open-weight models, directly applying orthogonalization edits to attention output projections and feedforward down-projections (Section 2.3).
     • Victim systems: three open-source aligned models; in the defense setting, a third-party abliterated checkpoint whose editing process is unknown to the authors, who operate only from the weights (Section 5.5).
     • Success criterion: a guard judge outputs a binary safe/unsafe label for each (prompt, response) pair, and ASR is the proportion of unsafe on the test set (Section 2.4).

   Collapse this answer

2. 02What related research exists?

   Related work unfolds in three categories: mechanistic studies of refusal directions and editing tools, precedents for weighted aggregation, and the optimization methods adopted in this paper.

   Mechanistic studies of refusal and editing tools

   • Arditi et al. [5] (2024): Propose that refusal is mediated by a single direction in the residual stream, that ablating with a difference-of-means vector can disable refusal, and establish the equivalence between activation ablation and weight-space orthogonalization; this paper's Basic RD baseline and intervention operators both come from this work.
   • Heretic (Weidmann [6], 2025): A public toolkit that automates [5]'s single-direction recipe end to end and uses an internal Bayesian search to tune per-layer strength; this paper directly adopts its intervention operator and fixed strength schedule and uses it as a baseline.
   • Wollschläger et al. [10] (2025): Propose RDO (learning a single refusal direction directly with gradient descent) and RDO-CONE (arguing that refusal is mediated by a cone of directions rather than a single line); the paper states that RDO-CONE is the prior construction closest in motivation to WRDO, with the difference that RDO-CONE learns a set of directions spanning a cone and ablates the entire cone, whereas WRDO fixes a per-prompt direction library and learns only the aggregation.

   Precedents for weighted aggregation

   • Ahsen et al. [7] (2019), FedWAPR [8], Weight-LoRA [9]: Use performance-aware weighted aggregation in ensemble classifier ranking, federated learning, and adapter selection, respectively; the paper cites them to support the motivation that weighting is better than uniform averaging when component quality is uneven, without comparing against them individually.

   Optimization methods

   • CMA-ES (Hansen & Ostermeier [11]; Hansen [12]): A gradient-free continuous black-box optimizer, adapted in this paper as WRD-CMAES.
   • NES / ES (Wierstra et al. [13]; Salimans et al. [14]): The evolution-strategies family; the paper states that both WRDO instances belong to this family, and that WRD-GRPO-IS can be read as ES with importance-mixing correction, with the difference that it routes scalar fitness through token-level policy ratios and can carry a KL term.

   Baselines and benchmarks

   • Baseline methods: Basic RD, Heretic, RDO, RDO-CONE, plus the sanity-check baseline Uniform-avg WRD constructed in this paper (same candidate library, weights fixed to a uniform vector).
   • Benchmarks/datasets: The harmful prompt pool is a deduplicated merge of six public benchmarks: AdvBench, MaliciousInstruct, TDC2023, HarmBench, JailbreakBench, StrongREJECT; the harmless baseline is a subsample of Alpaca; external evaluation uses the official HarmBench classifier; utility benchmarks are ARC-easy, ARC-challenge, MMLU, GSM8K, and WikiText perplexity.

   The authors position this paper as: between [5]'s single-direction recipe and [10]'s cone construction, proposing a third path of "fixed direction library + learning only aggregation weights," and comparing the two optimization routes, ES and RL, under the same objective and the same budget (Section 1, 3.2).

   Collapse this answer

3. 03How does the paper address this problem?

   WRDO constructs a per-layer direction library for each harmful prompt, learns a coefficient vector to aggregate them into a composite direction, and then writes it into the model via Heretic-style weight orthogonalization.

   Overall idea: change "find one refusal direction" into "find a set of aggregation weights"—first construct a candidate library of per-layer directions for each harmful training prompt, then use black-box optimization to learn a coefficient vector w that aggregates the library into a composite direction, and finally complete a one-shot edit with Heretic's weight-space orthogonalization operator.

   Candidate library and weighted direction

   • Per-prompt directions: for each harmful training prompt xi, take at each layer the difference between the residual-stream activation (at the last token position) and the harmless mean and L2-normalize it, r_i^(l) = (h^(l)(x_i) − μ_benign^(l)) / |h^(l)(x_i) − μ_benign^(l)|₂, i.e., each prompt has one unit direction at each layer (Equation 1).
   • Candidate library: the library R is computed once per model and shared by all optimization methods; size N = 128 prompts, sampling seed 42 (Appendix C).
   • Weighted composite direction: r^(l)(w) = ∑_i w_i r_i^(l) / |∑_i w_i r_i^(l)|₂; the scalar weights are shared across layers and coordinates, followed by per-layer L2 re-normalization, so the overall positive scale of w cancels out and only the orientation of the per-layer composite direction can vary (Equation 2).
   • Constraints: w is signed real-valued, and the feasible region is only the per-coordinate box [−2.0, +2.0]^N, with no simplex, norm, or sparsity constraints; all gradient-free methods are initialized at the uniform-average point wi = 1/N (Appendices A, F).

   Intervention operator and objective

   • Intervention: Heretic's weight-space orthogonalization is used, replacing each attention output projection and feedforward down-projection with W′ = W − α(l) r(w) r(w)⊤W (Equation 3); the only difference from Heretic is replacing the single direction with the composite direction of Equation 2.
   • Strength schedule: α(l) decays linearly from the peak position (max_weight_position = 0.7) to the window boundaries on both sides (min_weight_distance = 0.3, αmin = 0), and only the peak α ∈ {1.0, 1.5, 2.0, 3.0} is varied throughout (Appendix B).
   • Optimization objective: each guard outputs a binary verdict J(x, y), ASRJ is the proportion of unsafe on the validation distribution, and the objective is LJ(w) = −ASRJ(w); the attack minimizes LJ (Equations 4–6).

   Two optimizers

   • WRD-CMAES: optimizes LJ with Optuna's separable CmaEsSampler; each black-box query is a full WRDO evaluation (construct the composite direction, intervene on the model, measure ASR on the validation batch); budget n_trials = 50, seed 42, consuming the unregularized pure ASR signal (Section 3.2, Appendix F).
   • WRD-GRPO-IS: treats the objective as a non-differentiable function and optimizes it with reinforcement learning; adds a KL penalty β·KLw to LJ (the KL of the intervention policy relative to the clean model on the harmless batch), β = 1000.0 (Equation 8, Appendix F).
   • GRPO-IS mechanism: each step samples from M = 8 behavior policies, constructed by adding Gaussian noise εm ∼ N(0, σ²IN) (σ = 1) to the current w; because the samples are off-policy relative to the old policy, a sequence-level importance weight vem = πθold(ym|x)/µm(ym|x) (clipped) is introduced for correction, and the same weights are used to compute the group baseline; only w is updated, while the candidate library, base parameters, and α(l) stay fixed throughout (Equations 9–11).
   • Training parameters: AdamW, learning rate 10⁻³, n_epochs = 50, clip_ratio = 0.2, is_clip_ratio = 5.0 (Appendix F, Table 7).

   Success criterion and defense mode

   • Criterion: three guard judges (WildGuard, Qwen3Guard-Gen-8B, Llama-Guard-3-8B) each give a binary label, and ASR is reported per judge; the official HarmBench classifier serves as an external evaluator that never participates in optimization (Section 4.3).
   • Defense mode: the same pipeline and the same α grid with only the sign reversed, used to restore refusal on an abliterated checkpoint; for Basic RD this matches the defense recipe of [5], while Heretic and RDO do not expose a defense mode and therefore do not participate in that experiment (Section 5.5).

   Collapse this answer

4. 04What experiments did the paper conduct?

   Six methods are compared on three 7–8B open-source models and three guard judges; the authors report that WRDO outperforms single-direction baselines in both attack and defense, and that WRD-GRPO-IS preserves utility best.

   Experimental setup

   • Models tested: Qwen3-8B-Base, OLMo-3-1025-7B, Olmo-3-7B-Instruct; HarmBench evaluation additionally uses Qwen3-8B; the scale comparison additionally uses the 21B gpt-oss-20b (3.6B active parameters); the defense setting uses the third-party abliterated checkpoint Orenguteng/Llama-3.1-8B-Lexi-Uncensored-V2 (Sections 4.2, 5.4, 5.5, Appendix I).
   • Data: six public harmful benchmarks deduplicated and merged, pre-split into train/validation/test; the candidate library takes N = 128 from the training set; the harmless baseline is the first 400 examples of the mlabonne/harmless_alpaca training set, and the KL penalty uses 50 examples from its test set (Section 4.1, Appendix C).
   • Baselines: Basic RD, Heretic, RDO, RDO-CONE, Uniform-avg WRD (same library, uniform weights, no optimization).
   • Metrics and judges: the main metric is test-set ASR per judge; the judges are WildGuard, Qwen3Guard-Gen-8B, and Llama-Guard-3-8B, each outputting binary safe/unsafe; external evaluation uses cais/HarmBench-Llama-2-13b-cls; utility is measured with lm-evaluation-harness on ARC-e, ARC-c, MMLU, GSM8K and WikiText perplexity (Sections 4.3–4.5).
   • Decoding and repetition: greedy decoding, at most 100 new tokens, one response per prompt; each intervention unit in Table 1 is the mean and standard deviation of three independent full-pipeline repetitions (Section 4.2, Table 1).
   • Optimization signal: the utility and HarmBench experiments use only the w⋆ obtained by optimizing against WildGuard (Sections 5.3, 5.4).

   Main results

   Attack-setting test-set ASR (per Table 1, α ∈ {1.0, 3.0}, judges WG/Q3G/LG3):

   Swipe horizontally to read the table

   Model	Method	WG α=1.0	Q3G α=1.0	LG3 α=1.0	WG α=3.0
   Qwen3-8B-Base	Basic RD	0.22	0.11	0.44	0.81
   Qwen3-8B-Base	Heretic	0.77	0.83	0.81	0.77
   Qwen3-8B-Base	RDO	0.34	0.41	0.48	0.34
   Qwen3-8B-Base	WRD-GRPO-IS	0.91	0.94	0.87	0.93
   OLMo-3-1025-7B	Basic RD	0.41	0.51	0.43	0.76
   OLMo-3-1025-7B	WRD-GRPO-IS	0.82	0.86	0.79	0.92
   Olmo-3-7B-Instruct	Basic RD	0.40	0.42	0.43	0.53
   Olmo-3-7B-Instruct	WRD-GRPO-IS	0.92	0.88	0.93	0.93

   • The table omits the Uniform-avg WRD and WRD-CMAES rows and the Q3G/LG3 columns at α=3.0; the full grid is in Table 1, and intermediate strengths α ∈ {1.5, 2.0} are in Table 5 of Appendix D.
   • The authors report that both learned WRDO instances outperform all prior single-direction baselines by a clear margin in almost every configuration; Uniform-avg WRD is consistently below the learned instances across the entire α grid, from which the authors conclude that the gains come from learning the aggregation weights rather than from the candidate library itself (Section 5.1).
   • Exception: Appendix I reports that RDO-CONE leads WRD-GRPO-IS on Qwen3-8B-Base by 3.3 ASR points, 0.953 vs 0.93, while WRD-GRPO-IS leads by 5.5 and 8.5 points on the two OLMo checkpoints (Table 12, single run).

   Utility preservation

   • What was measured: four accuracy benchmarks and WikiText perplexity were run on the same w⋆ that produced Table 1 (Table 2).
   • Results: at α=3.0 on Qwen3-8B-Base, Basic RD's GSM8K drops from 0.84 to 0.00, MMLU drops to 0.23, and PPL exceeds 10¹²; WRD-CMAES's GSM8K drops to 0.004; WRD-GRPO-IS maintains GSM8K 0.83, ARC-e 0.64, and PPL 11.35 (the clean model is 11.11).
   • Authors' interpretation: WRD-GRPO-IS preserves utility best because its surrogate objective includes a KL regularizer; WRD-CMAES optimizes only the pure ASR signal and shows the same capability degradation as prior baselines; Heretic's KL objective preserves perplexity but still costs 40 MMLU points and 65 GSM8K points on Qwen3-8B-Base (Section 5.3).

   HarmBench external evaluation

   • What was measured: ASR of the instruct variants (Qwen3-8B, Olmo-3-7B-Instruct) was re-evaluated with the official HarmBench classifier, which never participates in optimization (Table 3).
   • Results: on Qwen3-8B, WRD-GRPO-IS reaches 0.97 at α=3.0 and WRD-CMAES reaches 0.97 at α=2.0; Heretic is fixed at 0.87, Basic RD at most 0.48, and RDO 0.39.
   • Authors' interpretation: the guard-judge metrics agree with the out-of-training classifier, from which the authors conclude that WRDO's advantage is a real effect rather than overfitting to the training signal (Section 5.4).

   Defense setting

   • What was measured: the same pipeline with the sign reversed was applied to the third-party abliterated checkpoint Llama-3.1-8B-Lexi-Uncensored-V2, measuring ASR under three judges (Table 4) and the external HarmBench re-evaluation (Table 11).
   • Results: the undefended checkpoint has ASR about 0.85 (three judges) / 0.94 (HarmBench); at α=−1.5 both WRDO instances lower the three-judge ASR to 0.011–0.073 and HarmBench ASR to 0.02–0.03; Basic RD's HarmBench ASR remains above 0.90 until α=−3.0.
   • Authors' interpretation: the authors report that both learned instances clearly win in the defense setting and that Basic RD fails to restore refusal; the same mechanism adapts to defense by reversing the sign, with the optimizer and candidate library unchanged (Section 5.5).

   Other ablations and analyses

   • Cross-judge transfer (Appendix H, Table 9): interventions optimized against WildGuard transfer reasonably well to the other two judges, while the reverse is unstable; the authors consider WildGuard the most reliable optimization signal among the guards considered.
   • β sweep (Appendix K, Tables 10, 13): sweeping β ∈ {1, 10, 10², 10³, 10⁴} on Qwen3-8B-Base at α=1.0, β=10³ is the only value that both maintains ASR 0.91 and matches the clean checkpoint's utility, and it is reused for all models and judges after selection.
   • Scale comparison (Appendix I, Table 12): repeating the attack setting on gpt-oss-20b, WRD-CMAES 0.923 and WRD-GRPO-IS 0.931, above Heretic 0.772 and Basic RD 0.308 (single run).
   • Runtime (Appendix G, Table 8): on a single H100 80GB, WRD-GRPO-IS takes about 11 hours, WRD-CMAES about 4 hours, and Basic RD about 2 hours.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note two limitations: model-scale coverage and the lack of KL regularization in WRD-CMAES; the experiments cover three 7–8B open-source models and one 21B MoE.

   Limitations and future directions noted by the authors

   • Model scale: Section 7 notes that the full experimental protocol covers only three open-source checkpoints of similar scale, all in the 7–8B parameter range.
   • WRD-CMAES without KL regularization: Section 7 notes that the current WRD-CMAES implementation consumes only the pure ASR objective of Equation 5 and does not yet support the KL regularization of Equation 8; the authors state they are developing a KL-aware sampling objective exposed in the Optuna interface and plan to add it to the codebase in a future version.

   Scope of experimental coverage

   • Models evaluated: The main protocol covers three 7–8B open-source models: Qwen3-8B-Base, OLMo-3-1025-7B, and Olmo-3-7B-Instruct; the HarmBench evaluation adds Qwen3-8B; the scale comparison adds the 21B gpt-oss-20b (Appendix I, single run); the defense setting uses a third-party abliterated Llama-3.1-8B checkpoint (Sections 4.2, 5.4, 5.5).
   • Data scale: The harmful pool is a deduplicated merge of six public benchmarks; the candidate library is fixed at N = 128 training prompts; the harmless baseline is 400 prompts, and the KL batch is 50 prompts (Appendix C); the paper does not report the total number of training/validation/test examples after deduplication.
   • Evaluation method: Three guard judges plus the official HarmBench classifier, all automated binary decisions; the paper does not report human evaluation or agreement between judges and humans.
   • Number of repetitions: The intervention results in Table 1 are the mean of three independent repetitions; the behavior of RDO-CONE and gpt-oss-20b is a single run (Table 12); the Abliterated behavior in the defense setting is evaluated once (Table 4).
   • Optimization budget: WRD-CMAES has a query budget of n_trials = 50, and WRD-GRPO-IS uses 50 epochs with 8 behavior policies per epoch (Appendix F); the utility and HarmBench experiments evaluate only the w⋆ optimized against WildGuard (Sections 5.3, 5.4).

   Collapse this answer

6. 06Summarize the main content of the paper

   WRDO constructs a composite intervention direction by learning aggregation weights over per-prompt refusal directions; the authors state that it outperforms single-direction baselines in attack, utility preservation, and defense.

   An attack paper on intervening in the refusal mechanism of aligned models: the authors argue that the standard difference-of-means single-direction construction mixes heterogeneous harmful prompts into one mean, and propose WRDO—constructing a candidate library of per-layer directions for each harmful training prompt (N = 128), learning a coefficient vector w to aggregate them into a composite direction, then editing the model once with Heretic's weight-space orthogonalization operator; the optimizers are WRD-CMAES (pure ASR objective) and WRD-GRPO-IS (a GRPO importance-sampling variant with a KL regularizer).

   • Attack effectiveness: on Qwen3-8B-Base, OLMo-3-1025-7B, Olmo-3-7B-Instruct and three guard judges, the authors report that the two WRDO instances outperform Basic RD, Heretic, and RDO in almost all configurations; for example, on Olmo-3-7B-Instruct at α=1.0, WRD-GRPO-IS achieves ASR 0.92/0.88/0.93 (WG/Q3G/LG3) versus Basic RD's 0.40/0.42/0.43 (Table 1).
   • External validation: on the HarmBench classifier (not involved in optimization), WRD-GRPO-IS on Qwen3-8B reaches 0.97 at α=3.0, Heretic 0.87, and Basic RD at most 0.48 (Table 3); the authors take this to mean the advantage is not overfitting to the training judge.
   • Utility preservation: at α=3.0 on Qwen3-8B-Base, WRD-GRPO-IS maintains GSM8K 0.83 and PPL 11.35, whereas Basic RD's GSM8K drops to 0.00 and PPL exceeds 10¹², and WRD-CMAES's GSM8K drops to 0.004 (Table 2); the authors attribute this to the KL regularizer in the GRPO-IS objective.
   • Defense use: with the sign of the same mechanism reversed and applied to a third-party abliterated checkpoint, at α=−1.5 it lowers the three-judge ASR from about 0.85 to 0.011–0.073 and HarmBench ASR from 0.94 to 0.02–0.03, while Basic RD fails to restore refusal before α=−3.0 (Tables 4, 11).
   • Exceptions and boundaries: RDO-CONE leads WRD-GRPO-IS on Qwen3-8B-Base, 0.953 vs 0.93 (Table 12); the authors note that experiments cover only 7–8B open-source checkpoints and that WRD-CMAES does not yet support KL regularization (Section 7).
   • Authors' conclusions: they recommend WRD-GRPO-IS optimized against WildGuard as the default configuration for auditing and defense; they argue that refusal behavior may be concentrated in low-dimensional internal features and is fragile under targeted intervention, that refusal should not be treated as a complete safety mechanism, and that deployment requires layered defenses (Sections 6, 9).

   Collapse this answer

Related papers