Has Abliteration Become Harder? Not Consistently: Direction-Based Ablation and ARA for Refusal Removal in Qwen3.8, Gemma 4, Olmo 3.1 and Mistral Small 3.2 LLMs and Their Predecessors in Heretic
Is Abliteration Getting Harder? Not Uniformly: Refusal Removal by Direction-Based Ablation and ARA in Heretic on the Qwen3.8, Gemma 4, Olmo 3.1 and Mistral Small 3.2 LLMs Versus Their Predecessors
The authors uniformly evaluated refusal removal on five old-new version pairs of open-source models: the authors state that removal did not consistently become harder, and that the effect depends on the version pair, method and budget.
The attacker uses public weights to perform fine-tuning-free weight editing (abliteration/ARA) on open-source instruct models, aiming to minimize dev keyword refusals under a budget of first-token KL ≤ 0.1 and 100 TPE search trials; the victim systems are instruct models such as Qwen, Gemma, OLMo, and Mistral Small, and success is measured by residual refusals as judged by WildGuard on 972 held-out harmful prompts.
- Refusals in open-weight models can be removed by fine-tuning-free abliteration edits; the ARA authors claim directional ablation is increasingly ineffective on new models, but this claim previously lacked independent testing, and tools commonly self-evaluate with keyword counts and first-token KL, which may disagree with stronger judges.
- On four main model pairs—Qwen, Gemma, OLMo, and Mistral Small—and one small Qwen pair, DiM, MPOA, SOM, and ARA were run uniformly with Heretic 2.0 (the small pair additionally with J-space), using 100 TPE trials, first-token KL ≤ 0.1, and the same selection rule; WildGuard was then used to evaluate residual refusals on 972 held-out harmful prompts, and capability and distribution shift were also measured.
- The authors report that removal did not consistently become harder: the old-new residual refusal difference for direction-class methods ranges from −1.2 pp for Mistral Small to +36.9 pp for OLMo, and for ARA from −26.0 pp for Gemma to +34.4 pp for Qwen; only the OLMo direction-class increase exceeds run-to-run variation under both variance models; keyword counts agree with the judge on average in the tool window but differ by about 7 pp per model.
- The authors note: the main study has only one optimization seed per cell, and run-to-run variance is transferred from the small pair; each pair has only one ARA run and export requires refitting; there is no human-label audit, and WildGuard was not validated on the edited models; only greedy decoding was used, with reasoning disabled; generation is confounded with scale, architecture, and training recipe.
- Evaluated models
- Qwen2.5-32BQwen3.8-27BGemma-2-27BGemma-4-31BOLMo-2-32BOlmo-3.1-32BMistral-Small-2409Mistral-Small-3.2Qwen2.5-7BQwen3.5-4B
- Benchmarks
- JailbreakBenchXSTestStrongREJECTHarmBenchcode-as-weapon bankMMLU-ProGSM8KIFEvalHumanEvalMBPP
- Metrics
- residual refusal (WildGuard)generation effect (pp)H2 contrast (pp)first-token KL divergenceteacher-forced divergencekeyword refusalCohen's κpass@1exact match
An arXiv paper, after unified conditional testing, found that refusal removal (abliteration) in open-source LLMs does not consistently become harder with model updates; whether it is harder to remove depends on the release pair, the method and the search budget, rather than on the model generation itself. In Heretic 2.0, the authors used direction methods such as difference-in-means, MPOA and SOM, along with ARA, on five old-new version pairs of Qwen, Gemma, OLMo and Mistral Small, performing fine-tuning-free weight editing under the same prompts, nominal search budget and first-token KL divergence cap, with a WildGuard judge determining refusals on 972 held-out harmful prompts (including benchmarks such as JailbreakBench and the authors' own malicious code library). After removal by direction methods, the new versions mostly retained -1 to +37 percentage points of refusal, and after ARA, -26 to +34; after accounting for run-to-run variance, the between-group heterogeneity I² of the direction-method effect is between 20-80%.
Paper analysis
What problem does this paper attempt to solve?
Tests whether newer-generation open-source models are harder to remove refusals from via abliteration; the authors state that the answer is not that it consistently became harder.
Core question: whether refusals in newer-generation open-weight LLMs are harder to remove via fine-tuning-free weight editing (abliteration) than in the previous generation.
- Setting and importance: safety training of open-source models can be edited after the fact via public weights; abliteration requires only a few hundred contrastive prompts and a single GPU, and Hugging Face already hosts many abliterated models and fully automated tools, which the authors consider relevant to attackers, defenders and auditors.
- Existing claims and shortcomings: the ARA authors claim that older models are "usually very easy to uncensor with standard abliteration," whereas directional ablation on the latest-generation models is "often far from sufficient"; the authors note that this claim had not previously been independently tested, that ARA has no published evaluation, and that existing abliteration comparisons predate ARA and current models.
- Methodological gap: tools typically evaluate with their own optimized cheap signals (keyword refusal counts, first-token KL), and evidence cited by the authors shows that keyword matching agrees poorly with human labels and that conclusions may depend on the classifier used.
- What the paper proposes: on five old-new version pairs, compare direction-class methods (DiM, MPOA, SOM, plus J-space on the small pair) with ARA using the same framework, prompt set, nominal search budget, KL cap and selection rule, and measure residual refusal on held-out prompts with a WildGuard judge independent of the optimization objective; the authors call it the first study they know of that independently compares ARA and direction-class methods across versions.
- Threat model:
- Goal: remove the refusal behavior of open-source instruction models so that they respond to harmful requests.
- Knowledge: the attacker has the model's public weights (open-weight) and can run and edit it locally.
- Capability: fine-tuning-free weight editing, with a budget of 100 TPE search trials, first-token KL ≤ 0.1, selecting the trial with the fewest dev keyword refusals.
- Victim systems and success criterion: instruction models such as Qwen, Gemma, OLMo and Mistral Small; success is measured by the residual refusal rate judged by WildGuard on 972 held-out harmful prompts, with higher residual refusal meaning "harder."
What related research exists?
Related work is divided into four categories by refusal-geometry assumption, plus three groups of literature on post-training mechanisms, defenses, and evaluation issues.
Single-direction ablation
- Arditi et al. (2024)—causally verified on 13 chat models that refusal is mediated by a single residual-stream direction; adding this direction induces refusal and ablating it suppresses refusal; subsequent orthogonalization of weights yields fine-tuning-free edits requiring only a few hundred prompts.
- Projected abliteration and MPOA (Lai, 2025)—respectively project out the harmless-mean direction and restore row norms to reduce collateral damage; Heretic (Weidmann, 2025) automates layer, strength, and direction selection with TPE. The paper notes that these variants are mainly documented in blogs, and that related preprints evaluate models no larger than 14B.
Multi-direction and subspace methods
- Gabliteration (Gülmez, 2025), OBLITERATUS (2026), WRDO (Evseev et al., 2026)—project multiple singular vectors or learn per-prompt direction weights.
- Wollschläger et al. (2025)—learn refusal directions with gradients and find a multi-dimensional "concept cone"; Piras et al. (2026) show that SOM generalizes difference-in-means, and that ablating multiple SOM directions outperforms single-direction baselines. The paper also notes that this premise is disputed: Joad et al. (2026), Wang et al. (2026), and others find that steering with different refusal directions yields nearly identical trade-offs.
Optimization-based methods
- ARA (Weidmann, 2026)—fits a low-rank update to each target module so that harmful outputs move closer to harmless outputs while harmless outputs are kept unchanged; its authors claim directional ablation is increasingly ineffective on new models and that ARA is better on gpt-oss-20b, and the paper notes that this evidence comes from a pull request and that the nearest-neighbor objective may overfit its prompt set.
Representation-space methods
- J-space ablation (Sekinal, 2026)—computes the refusal direction in the workspace defined by the Jacobian lens and pulls it back to the residual stream; the paper notes that by its authors' own measurements it removes very few refusals, and that the reference code is a single-model forward hook.
Post-training mechanisms, defenses, and evaluation
- Mechanism studies: DPO retains toxic capabilities and learns to bypass (Lee et al., 2024); safety fine-tuning makes minimal transformations to MLP weights (Jain et al., 2024); self-repair occurs after ablation (McGrath et al., 2023).
- Defenses: adversarial training on refusal features reduces the Llama-3-8B attack success rate from 53% to 10% (Yu et al., 2025); fine-tuning with randomly ablated directions reduces attack success rate by about 95% (Xie et al., 2025); decoy directions reduce Heretic weight-level attacks from 88.7% to 18% (Muhamed et al., 2026).
- Evaluation issues: keyword matching has κ = 0.37 against human labels, WildGuard 0.61, and fine-tuned LLM judges about 0.81 (Xie et al., 2025); one unreviewed practice report claims keywords count 3–5 refusals per hundred while LLM judges count more than 33 (Heretic PR #196).
Baselines and benchmarks: this paper does not compare against external attack baselines; the comparison objects are DiM, MPOA, SOM, J-space, and ARA within the same Heretic 2.0, plus 8 public community ablation models as external references; the held-out evaluation sets are JailbreakBench, XSTest, StrongREJECT, HarmBench, and a code-as-weapon prompt library compiled by the authors.
Positioning: the authors say this paper differs from prior work in four respects—held-out testing with version pairs as the unit, four method classes sharing the same framework and budget, scoring with a harness independent of the optimization objective, and (the authors claim) the first independent cross-version comparison of ARA and directional methods; null-space ablation, refusal cones, WRDO, optimal-transport ablation, and fine-tuning attacks are outside the evaluation scope.
How does the paper address this problem?
Five old-new model pairs × four classes of editing methods, with a uniform 100-trial TPE search, KL ≤ 0.1 selection rule and independent judging harness.
Overall approach: cross "model generation" with "refusal-removal method," and compare residual refusals under the same selection rule, the same evaluation harness independent of the optimization objective, and the same nominal budget (100 trials); the method framework is Heretic 2.0.
Models and data
- Models: four main research pairs—Qwen, Gemma, OLMo, Mistral (22–32B, bf16 single GPU)—plus one small Qwen pair (7B→4B); each pair consists of an old and a new instruct model from the same family, with the new version released 9–23 months later (Table 2).
- Architecture disclosure: Gemma-2/4 and OLMo-2/Olmo-3.1 are post-norm, and directional editing is incomplete on these four models; the authors disclose this rather than correcting it in the main runs, and run a separate γ-MPOA control.
- Prompt data: direction estimation, SOM fitting, and ARA module capture used the first 400 training prompts each from mlabonne/harmful_behaviors and harmless_alpaca; the dev set was the first 100 test prompts from harmful_behaviors.
- held-out set: 1,322 prompts (972 harmful, 350 harmless) from JailbreakBench, XSTest, StrongREJECT, HarmBench, and a 200-prompt code-as-weapon library compiled by the authors (Table 3); frozen before any held-out responses were generated, and 44 entries overlapping with search prompts were flagged.
Method instantiation
- DiM: a rank-1 edit removes the difference-in-means direction, with direction, strength, and layer profile tuned by the same search as MPOA ("Heretic-tuned DiM").
- MPOA: the direction is projected orthogonal to the harmless mean, and row norms are restored after row normalization (rank-3 adapter).
- SOM: a port of the proof of concept from Heretic PR #196; a 4×4 SOM per layer takes the k = 4 neuron directions with the most wins, which are orthonormalized and ablated with a searched geometric decay (rank-6 adapter).
- ARA: a rank-50 adapter is fitted to each target module, optimizing with L-BFGS the objective L = wp |Ỹg − Yg|2 + ws dk(Ỹb, Yg) − wo dk(Ỹb, Yb), meaning: keep harmless outputs unchanged, pull harmful outputs toward harmless outputs, and push them away from their original position (a reverse term to prevent over-correction); row-norm preservation enters only the objective, and the exported adapter is no longer normalized.
- J-space (small pair only): the jspace-refusal code computes the direction from the Jacobian lens and the refusal-token unembedding and bakes it into the weights (rank-3), rather than the original authors' forward hook.
Search and selection
- Optimization: each model×method is an Optuna multivariate TPE study with 100 trials (first 30 random), seed 1; dual-objective minimization—the proportion of the 100 dev responses containing 33 refusal markers, and the first-token KL divergence on 100 harmless prompts.
- Selection rule: among completed trials with first-token KL ≤ 0.1, select the one with the fewest dev refusals, breaking ties by lower KL and then by trial number; sensitivity analyses were also done at thresholds of 0.05, 0.2, and 0.5.
- Export: the selected trial is merged into a standalone checkpoint, changing only the edited matrices; 42 main models and 4 γ-MPOA controls were released (except 5 Gemma-2-27B variants).
Evaluation and statistics
- Generation: one greedy generation per held-out prompt, up to 512 tokens, reasoning disabled, unified system prompt.
- Judging: the primary outcome is the WildGuard refusal label; also the HarmBench classifier, Qwen3Guard-Gen-8B, two encoder classifiers, and 33 keyword markers (full text and first 100/256 tokens); a degeneration heuristic flags empty, repetitive, or unprintable responses.
- Capability and shift: five benchmarks—MMLU-Pro, GSM8K, IFEval, HumanEval, MBPP (Table 4)—and teacher-forced divergence on 350 harmless prompts.
- Tests: paired McNemar exact test, Holm correction; H1 is the paired difference in residual refusals between new and old models on the same 972 prompts; H2 is the GEE generation×method-class interaction; run-to-run variance was estimated post hoc from 10 seed pairs on the small pair (σ = 9.2 pp), with both a constant variance model and a variance model that varies with refusal level.
What experiments did the paper conduct?
H1–H3 tests on five pairs: removal did not consistently become harder, and only the OLMo direction-class increase is robust under both variance models.
Experimental setup
- Models tested: Qwen2.5-32B→Qwen3.8-27B, Gemma-2-27B→Gemma-4-31B, OLMo-2-32B→Olmo-3.1-32B, Mistral-Small-2409→Mistral-Small-3.2, Qwen2.5-7B→Qwen3.5-4B (small pair), for 10 original models in total plus their respective edited versions.
- Methods: DiM, MPOA, SOM, ARA, with J-space added for the small pair; also 4 γ-MPOA controls, 10 seed-2 reruns (small pair only), and 8 community releases.
- Data: held-out 1,322 prompts (972 harmful, 350 harmless; Table 3); 100 dev prompts for search and selection.
- Metrics and judges: the primary metric is the residual refusal rate as judged by WildGuard on the 972 harmful prompts; the second judge is Qwen3Guard-Gen-8B, and the HarmBench classifier measures harmful compliance; no human audit.
- Budget and repetition: 100 TPE trials per cell, seed 1; the main study has only one optimization per cell, and the run-to-run SD of 9.2 pp is estimated from two seeds on the small pair.
- Statistics: exact McNemar + Holm (23 tests in total for H1); intervals are reported both for prompt sampling only and with run-to-run variance added.
Main results
New-minus-old residual refusal difference (newer − older, pp, WildGuard, 972 harmful prompts; per Table 5, Table 6):
Swipe horizontally to read the table
Version pair Original model Directional class (mean) ARA H2 comparison (ARA−directional) Qwen +13.1 +10.7 [−4.0, +25.5] +34.4 [+8.8, +59.9] +23.6 [−5.9, +53.1] Gemma +1.0 ns +4.7 [−10.2, +19.6] −26.0 [−51.6, −0.5] −30.8 [−60.2, −1.3] OLMo +0.8 ns +36.9 [+22.1, +51.7] −2.5 [−27.9, +23.0] −39.3 [−68.8, −9.9] Mistral Small +17.9 −1.2 [−16.0, +13.5] −0.7 [−26.1, +24.7] +0.5 [−28.8, +29.9] Small pair +16.4 +17.3 [+4.4, +30.1] +14.2 [−11.3, +39.7] −3.1 [−31.6, +25.4] Intervals are 95% intervals with run-to-run variance added; the Gemma directional class includes the failed Gemma-4-31B DiM edit (residual refusals no lower than its original model).
- H1 did not hold: the authors report that no main research pair had higher residual refusals for the new version across all methods; directional-class effects ranged from −1.2 pp (Mistral Small) to +36.9 pp (OLMo), and ARA from −26.0 pp (Gemma) to +34.4 pp (Qwen).
- Robustness: under the run-to-run SD point estimate, the only original per-pair effect whose interval excluded zero under both variance models was the OLMo directional increase; the Qwen and Gemma ARA effects no longer excluded zero under the variance model that varies with refusal level.
- Heterogeneity: the authors report that between-pair heterogeneity of directional-class effects varies from low to high across scales and variance models (I² of 20% on the log-odds scale, 80% on the pp scale with constant variance, and 46% with refusal-level variance).
H2: ARA versus directional methods
- What was measured: the new-minus-old effect difference between ARA and the directional-class mean for each pair (Table 5, Table 8).
- Results: ARA lost less effectiveness than directional methods on Gemma (−30.8 pp) and OLMo (−39.3 pp), and lost more on Qwen (+23.6 pp, interval includes zero, uncertain); the pooled comparison was −11.5 pp [−57.6, +34.6].
- Authors' interpretation: each comparison is based on only one ARA run, and conclusions change with the KL budget and variance model; the authors say support for the ARA authors' claim is limited to the Gemma and OLMo pairs and is conditional on these cells.
H3: Agreement between keyword counts and automatic judges
- What was measured: on the same held-out responses, the difference between Heretic's 33 keyword markers and WildGuard's refusal judgments (Table 9, Table 10, Figure 4).
- Results: the underestimation predicted by H3 did not hold—the 100-token window (used in Heretic search) had a mean difference of +0.5 pp [−1.0, +1.9], but per-model means differed by 7.0 pp, with 25 models differing significantly and in differing directions; the full-text window keywords instead over-counted by +8.4 pp. Agreement between keywords and WildGuard fell from κ = 0.73 for the original models to 0.40 after ablation (main study), and was 0.82 between the two LLM judges.
- Authors' interpretation: there is disagreement between two raters that both make errors, and the keyword-optimized search is guided by a signal whose relationship to refusal judgments varies by model; without human labels it cannot be determined who is correct.
Other ablations and analyses
- γ-MPOA control: after correcting for post-norm gain, residual refusals fell on all four post-norm models (Gemma-4-31B by 19.9 pp), and the OLMo generation effect remained +57.0 pp; Gemma-4-31B and OLMo-2-32B lost 21.4 and 8.1 pp of average capability, respectively (Figure 5).
- Community releases: 4 of 8 public ablations had lower residual refusals than the best selected model for the same checkpoint in this paper (e.g., 1.4% versus 14.0% on Qwen3.8-27B), with teacher-forced divergence of 0.085–0.151 (Figure 6).
- Item source: the Gemma directional increase came entirely from the code-as-weapon library (code items +49.3 pp, public sets −6.8 pp); the OLMo increase held on both source types (Table 7).
- Seed sensitivity: a second seed on the small pair significantly changed held-out refusals for all 10 cells (median 9.4 pp, maximum 25.2 pp) and erased the ARA generation effect (+14.2 pp for seed 1, +0.0 pp for seed 2) (Figure 7).
- Capability and shift: average capability change across the 42 main models ranged from −4.2 to +0.9 pp; capability change correlated with teacher-forced divergence (ρ = −0.48) but not with dev first-token KL (ρ = −0.03), on which basis the authors state that first-token KL cannot constrain capability loss.
- Harmful compliance: after ablation, 75.1% of responses in the main study not judged as refusals by WildGuard were judged harmful by the HarmBench classifier; with harmful compliance as the outcome the pooled conclusion is unchanged, but the Qwen pair meets the H1 criterion under HarmBench (Appendix D.6).
What are some points that can be further explored?
The authors list limitations including a single seed, a single ARA run, and no human audit, and give four directions for future work.
Limitations noted by the authors and future directions
- Single-seed optimization (§5.1): the main study used only seed 1 per cell; a second seed on the small pair changed the held-out refusal of all 10 cells (median 9.4 pp) and erased the ARA generational effect; the hypothesis that the 9.2 pp run-to-run SD transfers from 4B/7B models to 22–32B models could not be tested. Future direction: run a second seed for the 32 main-study cells (§6).
- ARA cell issues (§5.1): the released ARA models were reconstructed by refitting adapters; four exports refitted on a different GPU from the search drifted 5–9 refusals on dev rescoring; the Gemma-4-31B ARA export rescored at KL 0.107, exceeding the cap; both the Qwen reversal and the Gemma support for H2 rest on a single ARA cell, and the authors call the per-pair H2 determination preliminary. Future directions: run the Qwen3.8-27B ARA trial reported by heretic-org in this pipeline, refit exports on the search hardware, and run a second ARA seed for the Qwen and Gemma pairs (§6).
- No human audit (§5.1): all refusal rates are relative to automatic judges, and WildGuard was not validated on edited model outputs; where keyword and judge disagree, it cannot be determined which is right. Future direction: conduct a human-labeled audit of responses where keyword and judge disagree (§6).
- Generation settings (§5.1): a single greedy response per prompt, at most 512 tokens, with reasoning disabled; sampling decoding may substantially change refusal rates, and chain of thought may carry refusals; conclusions apply only to greedy, non-thinking mode.
- Generational confounding (§5.1): generation is confounded with scale, architecture, training data, post-training recipe and release interval; post-norm leakage puts direction-class methods at a disadvantage on 4 of the 8 main-study models, and DiM and SOM were not corrected; 200 author-written code entries account for about half of the original-model generational difference.
- Implementation scope (§5.1): J-space is limited to the small pair (only these two models have instruction-tuned Jacobian lenses), the direction is baked into the weights rather than a forward hook, and the direction of bias is unknown; Heretic-tuned SOM and Piras et al. have several deviations, and a narrower search may underestimate SOM; the paper-faithful SOM configuration and the J-space forward hook were planned but not run, and are left for future work. §6 also proposes: obtain judge labels for non-selected trials, and reselect the Pareto frontier with target and full-sequence divergence constraints.
Experimental coverage
- The models tested are 10 dense instruction models (4–32B) in 4 families and 5 pairs, with English prompts, a single Heretic 2.0 configuration, and 100 trials (Appendix C).
- Methods cover DiM, MPOA, SOM and ARA, with J-space only on the small pair; null-space ablation, refusal cone, WRDO and optimal-transport ablation were not evaluated, and fine-tuning attacks are out of scope (§2.7).
- The main study used one optimization seed per cell; the seed-2 rerun covered only the 10 cells of the small pair (§3.4).
- Judges are WildGuard (primary), Qwen3Guard-Gen-8B (second), the HarmBench classifier and two encoder classifiers; the paper does not report agreement between judges and human labels on edited models (§3.3, §5.1).
- External references are 8 community releases; results for the three OBLITERATUS presets on Qwen2.5-7B are reported in a sister paper, not in this one (§3.2).
Summarize the main content of the paper
A controlled comparison across five old-new pairs of open-source models: the difficulty of refusal removal depends on the version pair, method and budget, not on generation itself.
A controlled comparison study of refusal removal (abliteration) in open-weight models, testing the claim that "newer-generation models are harder to remove refusals from."
- Question: the ARA authors claim directional ablation is increasingly ineffective on the latest models, but the authors note this claim lacks independent testing, and that the keyword counts and first-token KL self-evaluation signals customary in tools may disagree with stronger judges.
- Method: on four main model pairs—Qwen, Gemma, OLMo, Mistral Small—and one small Qwen pair, DiM, MPOA, SOM, and ARA were run uniformly with Heretic 2.0 (the small pair additionally with J-space), using 100 TPE trials, first-token KL ≤ 0.1, and the same selection rule; WildGuard then judged residual refusals on 972 held-out harmful prompts.
- Main results: removal did not become consistently harder—the new-minus-old residual refusal difference for directional methods ranged from −1.2 pp for Mistral Small to +36.9 pp for OLMo, and for ARA from −26.0 pp for Gemma to +34.4 pp for Qwen (Table 5); under the run-to-run SD point estimate, only the OLMo directional increase exceeded run-to-run variation under both variance models.
- Method-class comparison: ARA lost less effectiveness than directional methods on Gemma and OLMo (−30.8, −39.3 pp), and the point estimate reversed on Qwen (+23.6 pp) but the interval included zero; each pair comparison is based on only one ARA run, which the authors call a preliminary comparison.
- Evaluation signals: the underestimation predicted by H3 did not hold—100-token-window keywords agreed with WildGuard on average (+0.5 pp) but differed by 7.0 pp on average per model; first-token KL could not constrain capability loss.
- Authors' conclusion: whether a model resists removal should be read as a statement about one version pair, one method, one budget, and one metric; the authors recommend that audits use at least two method classes, multiple seeds, held-out prompts from multiple sources, at least two judges, and multiple divergence budgets, and plan to run a second seed of the main study, a human-label audit, and a re-selected Pareto frontier.
Related papers
- TRACEAnalysis / interpretabilityarXiv
Show 3 more papers
- On Evaluating the Durability of Safeguards for Open-Weight LLMsAnalysis / interpretabilityICLR 2025