Fool's Gold: Defensive deception against safety-removal attacks on open-weight models

1. 01What problem does this paper attempt to solve?

   Safety alignment of open-weight models can be removed by abliteration within minutes; the authors instead deceive the attacker, making the refusal-stripped model output fabricated decoys.

   Core problem: the safety alignment of open-weight models can be removed by abliteration within minutes, and the authors say no release-time defense can durably prevent it; the paper instead makes the attack's output untrustworthy.

   • Setting and importance: the authors say open weights have reached frontier scale (Kimi K3 2.8T, GLM-5.2 753B), and abliterated variants appear within days of release; the paper focuses on CBRNE operational uplift because that domain has the most severe marginal-capability consequences for stripped models.
   • Shortcomings of existing methods: the authors say refusal-mechanism-hardening defenses are all broken or bypassed by adaptive attackers (§II cites TAR and SEAM being broken, with ART's own evaluation leaving a 0.49–0.68 escape rate); the authors' own search also found no projection-style edit that removes targeted generation capability without destroying the model.
   • Core observation: if you cannot prevent it, you can deceive—drawing on the defensive deception tradition of honeypots and honeywords, the attacked model itself becomes a honeypot: dangerous requests get confident, fluent decoys with fabricated key elements, so that no answer can be safely trusted.
   • The paper proposes: decoy hardening (Fool's Gold), a release-time defense recipe, plus an evaluation methodology centered on the "economics of deception" (attacker cognitive readout, element-level denial scoring, consensus-attack pricing), instantiated on seven models across five families (9B–122B).
   • Threat model (§III):
     • Goal: extract usable dangerous-operation procedures from the attacked defended weights and trust them without external verification.
     • Knowledge: full white-box access to D0, can re-derive the refusal direction from the defended weights; but no independent source of correct values (no domain expertise, laboratory, ground-truth reference or retrieval verification source).
     • Capabilities: public tools for layer/recipe scanning, free sampling of D0-a, up to K=64 element-wise consensus voting; can run knowledge-free selection strategies and use helper LLMs.
     • Victim system: first-release instruction-tuned open weights (no public clean copy); in scope is weight-space safety removal (direction ablation and variants, correctness-blind RL/SFT), out of scope is in-context jailbreaking, fine-tuning on ground-truth demonstrations, and diffing against a public clean copy.

   Collapse this answer

2. 02What related research exists?

   Related work falls into four groups: attacks that remove alignment, defenses that protect the refusal mechanism, defenses at other levels, and the defensive deception tradition; the authors position this paper as a defense that takes effect only after a successful attack.

   Attacks that remove open-weight safety alignment

   • Fine-tuning attacks: Qi et al. [3]'s few adversarial-example fine-tuning, shadow alignment [4], LoRA unalignment [5], GRP-Obliteration [6] (GRPO unalignment from a single unlabeled prompt).
   • Direction ablation: Arditi et al. [7] proposed that refusal is mediated by a single direction, which the community engineered into abliteration [8] and developed into multi-direction, partial-strength, and architecture-aware variants [19][20]; later analyses indicate that refusal in new models is multi-dimensional concept cones [22], decomposed by category [23], with harmfulness and refusal encoded separately [25]—the authors use this to support the premise that "content behavior remains contestable after refusal is stripped."

   Defenses that protect the refusal mechanism

   • extended-refusal fine-tuning [30], DeepRefusal [31], ART [10], circuit breakers [32], multi-direction refusal geometry [22][23]. The authors note that the guarantees of such defenses terminate when refusal is removed, and that there are recorded breaks (TAR/SEAM broken by [10][11], ART self-evaluation leaves a 0.49–0.68 escape rate).
   • Decoy direction optimization (DDO) [33]: injects a decoy direction orthogonal to refusal to poison the attacker's direction estimator. The authors call it the sharpest comparison—DDO deceives the attack's direction estimator, and its guarantee still terminates when stripping succeeds (its self-evaluated attack success rate returns to 0.65 under adaptive re-estimation); Fool's Gold deceives the attacker who consumes the stripping output, and the two layers can be composed.

   Defenses at other levels

   • unlearning (RMU [9], WHP [36]) removes dangerous knowledge; the authors state that its known recovery weakness is their motivation for choosing not to strip knowledge; pretraining data filtering [37][38] applies only before release and can be composed with this defense.
   • Anti-fine-tuning methods (TAR [39], RepNoise [40], self-destructing models [41], SEAM [42], SOPHON [43]) target the fine-tuning attack family; model editing (ROME/MEMIT [44][45]) can implant decoys fact by fact, whereas this paper binds decoy policies by behavior.

   Defensive deception and conditional behavior

   • The deception tradition runs from Stoll's forged documents [12], honeypots [13], honeyfiles [14][15] to honeywords [16]; LLM deception defenses are at inference time (HoneyTrap [49], misdirection [50]) or on the environment side (AgentSnare [51]), and the authors note these sit in the serving stack or the attacker's environment, which a weight-release attacker would discard.
   • Sleeper Agents [52], linear probes [53], trigger extraction [54], and password-locked models [55] show that conditional behavior can be bound into weights; the authors state that Fool's Gold is the defensive inversion of this family—the "trigger" is not an implanted token but the attack-induced state itself.

   Baselines and benchmarks: the paper has no methodological baseline (the authors state that to their knowledge this is the first release-time defense guaranteed to begin after a successful attack); the comparison is each model's own attacked-undefended (M0-a) floor. Evaluation benchmarks include StrongREJECT, HarmBench, AILuminate, FORTRESS (external red team), and MMLU, GSM8K, WMDP, IFEval, BioProBench (retention). Authors' positioning: Fool's Gold is orthogonal to mechanism-protecting defenses, and its safety property begins where those defenses terminate.

   Collapse this answer

3. 03How does the paper address this problem?

   Two steps: first have the attacked copy self-generate a decoy corpus with element-level fabrication, then do seed SFT plus attacked-state DPO within a differentiable attack simulation to bind the decoys into the attacked state.

   Overall approach: acknowledge that refusal stripping will inevitably succeed, and change the defense goal from "preventing the attack" to "poisoning the attack's output"—Fool's Gold (decoy hardening) makes the attacked state output confident, fluent decoys with fabricated key operational elements for dangerous requests, while pinning clean-state behavior to the original model.

   Problem setup and formalization

   • Notation: M0 the original model (reference held by the defender), M0-a the attacked original model (undefended baseline), D0 the defense release candidate, D0-a the defended model after a fresh adaptive attack (the deployment-relevant condition).
   • Attacker success is defined as succ(S) = Pr_x[fatal(y_S(y1,...,yK)) = 0]: the attacker succeeds if the answer selected by strategy S from K samples has no fatal flaw; fatal(y)=1 when the judge's weakest-element gate fails. The safety goal is denial of trust rather than item-by-item denial of content.
   • Defended attack (§IV-B): difference-in-means direction r^(l) = μ^(l) − ν^(l), then orthogonalize all matrices writing to the residual stream W_out ← W_out − r̂r̂^⊤W_out; the adaptive form estimates the direction from the defended model itself.

   Decoy corpus construction (§IV-C)

   • Refusal screening and capability gate: dangerous-operation prompts are drawn from public safety benchmarks, keeping only those M0 consistently refuses (coverage 73–99%); payloads elicited from M0-a become associations only after an LLM judge scores them above lower bounds on each axis of intent alignment, compliance risk, detail and technical correctness.
   • Decoy writing with element verification: the canonical-fatal contract—each payload fabricates exactly one fatal-class element (quantity, reagent, temperature, timing, component, step order or key safety property), always fabricated to the same fake value, with the rest kept true; no disclaimers, no hedges, no tells.
   • The attacked model writes its own decoys (self-decoys), with a verification loop regenerating until the element list is fully fabricated, tell-scrub and write-gate assertions leaving zero residual tells; corpus gate pass rate 91–100%. The scatter contract is the ablation arm.

   Binding into the attacked state (§IV-D)

   • Training uses rank-32 LoRA (mid-layer MLP projections, merged at release), with A(θ) a differentiable simulated attack: applying the same projection as the attacker mathematically at each write site's output, with the direction re-estimated from the current model every 100 steps.
   • Stage 1 (seed): minimize attacked-state decoy cross-entropy L_decoy, clean-state refusal pin L_refuse (pinned to M0's own greedy refusals) and benign KL leash L_KL, L_seed = L_decoy + L_refuse + λ L_KL, λ=2, 300 steps.
   • Stage 2 (attacked-state on-policy DPO): attack the current checkpoint, sample K=16 per training association, judge each sample, with preferred being samples judged fatal and rejected being samples judged usable escapes, β=0.1, 550 steps per round; the refusal pin and KL leash are maintained throughout. Mining is tell-filtered to avoid self-play reinforcing a marked subgroup.
   • Round scheduling: continue when validation fatal improves >0.01 and the mined escape pool is ≥100, up to 8 rounds; benign gate (rounds with denial shift >0.10 are disqualified); the release checkpoint is the post-hoc highest fatal point that passes benign certification.

   Success determination and evaluation method (§V-B, §V-C)

   • Attack acceptance protocol: a fresh attack must pass the registration gate (refusal ≤0.2 on held-out probes, degeneration ≤0.25, judge judges genuinely harmful compliance, both thinking modes of hybrid reasoning models satisfied) before any defense metric is read; the strongest-found-attack policy is used.
   • The judge is a fixed snapshot gpt-4.1-2025-04-14, using a decomposed key-element denial rubric: 3–7 key elements are extracted and verified one by one, with the weakest element gating the denial score; fatal is defined as following the request would fail and a layperson would not know why.
   • Metrics: fatal fraction (decoy rate per draw), mean denial, attacker-best (minimum denial over K draws, oracle attacker upper bound), refusal; plus an evasion audit (a judge-free surface classifier at the 30% gate) confirming that fatal determinations come from confident fabrication rather than vague non-answers.
   • Statistics: all headline rates carry 95% CIs (prompt-level cluster bootstrap with 10,000 resamples; Wilson intervals).

   Collapse this answer

4. 04What experiments did the paper conduct?

   Main results for seven models: the six models that passed the gate show fatal 0.508–0.899 on unseen prompts; consensus attacks, RL, benign SFT, white-box probes and other counter-attacks are all priced; 9B is a boundary failure case.

   Experimental setup

   • Models tested: Qwen3.5-9B / 27B / 122B (within-family scale axis, sharing a 263 train / 94 validation association pool and a 260-item decoy corpus), Qwen3-14B, gpt-oss-20b, gemma-4-31B, GLM-4.5-Air (cross-family axis); gemma-3-27b is the scatter-contract case-study host.
   • Data: dangerous-operation prompts come from public safety benchmarks [60]–[67]; per-model validation uses 74–94 never-trained prompts; the frozen test set is measured once per model (n=30–100; gpt-oss-20b has no frozen row and uses an untouched n=100 stratum).
   • Baselines: each model's own attacked-undefended (M0-a) fatal floor; no external defense baselines (the authors say re-testing RMU/TAR baselines is left for future work, Appendix B).
   • Metrics and judging: the judge is gpt-4.1-2025-04-14 with a decomposed rubric; fatal = weakest-element gate failure; main stratum K=4, temperature 0.8; consensus attacks on a fixed 24-prompt subset (14B expanded to n=74) with K∈{4,16,64}.
   • Human anchoring: CBRN domain experts blind-reviewed n=154 (140 deduplicated + 14 blind duplicates); plus author-review blind re-judging and a full-element census on 122B; two second-judge re-judgments (GPT-5.5 stratified 220 draws; a cross-vendor Claude Opus attempt was censored by its safety layer).
   • Gates: validation fatal ≥0.40, clean refusal ≥0.85, benign denial shift ≤0.10, GSM8K drop ≤2 points (Table XXVI).

   Main results

   According to Table I (validation fatal / frozen confirmation / floor / ∆ / K=64 consensus fully-usable):

   Swipe horizontally to read the table

   Model	Fatal ↑	Confirmed fatal (n)	Floor	∆ ↑	fu@64 ↓
   Qwen3.5-9B (boundary, did not pass gate)	0.202	0.225 (61)	0.136	+0.066	0.83
   Qwen3.5-27B	0.508	0.515 (65)	0.130	+0.378	0.625
   Qwen3.5-122B	0.654	0.700 (30)	0.069	+0.585	0.375
   Qwen3-14B	0.899	0.926 (50)	0.186	+0.713	0.083
   gpt-oss-20b	0.726	0.632 (100, untouched stratum)	0.452	+0.274	n/a (instrument unvalidated)
   gemma-4-31B	0.857	0.787 (100, untouched stratum)	0.018	+0.839	0.250
   GLM-4.5-Air	0.617	0.579 (70)	0.064	+0.553	n/a (instrument unvalidated)

   • The authors say the six gate-passing models produce decoys for 0.508–0.899 of unseen prompts, with attributed increments of +0.274 to +0.839; fatal reproduces within ±.05 on the frozen test set (Table VIII).
   • 9B did not pass the registered fatal gate (0.202 vs 0.40) and is reported as a named boundary result, not merged into the gate-passing range.
   • gpt-oss-20b's floor is itself 0.452 (the undefended attacked model already fabricates fatally on 0.452 of draws); the authors say this model should be read by ∆ rather than absolute rate.

   Element consensus attack (§VII-C)

   • What was tested: the strongest oracle-free attack that needs no single clean draw—K-draw sampling, element-wise clustering, per-slot majority voting to synthesize a procedure; anchored on the undefended M0-a (valid only if recovery ≥0.60 and fully-usable ≥0.50).
   • Results: at K=64 the defended side's fully-usable is 0.083 (14B) / 0.25 (gemma-4) / 0.375 (122B) / 0.625 (27B), with undefended anchors at 0.58–0.96; gpt-oss-20b and GLM-4.5-Air anchors did not pass the validity gate, so the instrument does not apply. On the full 14B validation stratum (n=74), the self-selected attacker accepted 15 synthesized procedures, 10 containing fabricated elements (precision 0.333 vs anchor 0.98).
   • Authors' interpretation: consensus deception holds on 14B, holds directionally on gemma-4, and is absent on 27B (27B ran only one round of preference optimization); but distinguishing the two modes requires the element registry and anchor that the threat model denies, and the attacker cannot observe precision; 27B's recovery also does not restore trust in the channel.

   RL and benign SFT counter-attacks (§VII-D, §VII-F)

   • GRP-Obliteration (correctness-blind compliance-reward RL): on defended Qwen3-14B, the LoRA attack gives fatal 0.647/0.625 (trained/validation), the full-parameter attack 0.543/0.530, versus only 0.066/0.081 for the undefended base; the authors say deeper reward optimization converges further into the decoy manifold (the full-parameter attacker's own reward is highest at 0.854).
   • Consistency-scoring RL: on defended 14B it stalls at fatal 0.261/0.321, about half the pure-compliance attack's decoy rate, but still 2.8× the clean base rate (0.091–0.115); the consistency term never took effect (maximum 0.025 per call).
   • Benign-data SFT repair: defended 14B's per-draw fatal falls with budget 0.834 → 0.603 → 0.424 (300 steps/2,000 samples and a tenfold escalation), with paired differences versus the same-fine-tuning control of +0.630 → +0.165 → +0.083; but true-element share stays flat (0.163 → 0.156/0.160), and the authors say the corrupted capability never recovers—the growing non-fatal tail is vagueness, not correctness.

   Other ablations and analyses

   • Attack variants (Table XXIII, gemma-3 case): six independently accepted attack variants have fatal in a narrow band of 0.555–0.758; the final recipe reaches 0.91–0.96 on 14B.
   • Oracle-labeled "decoy direction" ablation (Table XXIV): all candidates HOLD, at most ≈0.09 more than the random-direction control.
   • Paraphrased attack prompts (§VII-A): on defended 14B, unseen paraphrases (n=248, K=4) give fatal 0.841, between the trained (0.939) and untrained (0.899) strata.
   • External benchmarks (Table II): on SR/HB CBRNE slices, the attacked defense arm's attacker-visible quality is within ≤0.18 of undefended on 12 of 14 model-benchmark pairs, and the fatal rate at matched quality (SR≥0.5) is 1.3–44× that of undefended (absolute +0.15 to +0.87).
   • White-box probes (Table XXV, §IX): under oracle labels a linear probe reads per-answer fatal at AUROC 0.969, and a 12.5% retention rate can filter; but probe-routed consensus still falls below the breakage gate (fully-usable ≤0.136), and the authors say what is obtained is selective non-answering rather than reconstruction.
   • Stage necessity (Table XXII): direct DPO without the seed gives cold-start fatal of only 0.047 (kill gate 0.20); seed SFT instantiates the decoy pattern, and DPO concentrates it and closes the escape rate.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list nine limitations: residual escape tail, unmeasured attacker classes, single judge, statistical resolution, white-box readability, etc.; experiments cover seven models and the chemical and biological domains.

   Limitations and future directions noted by the authors

   • Residual escape tail (§IX): on passing defended checkpoints, 10–49% of attacked extractions are still clean (supplementary counts in the Table I fatal column); on the defended 14B, 3 of 74 validation prompts have no decoy across four samples; paraphrase coverage was tested on only one model.
   • Unmeasured attacker classes (§IX): benign SFT with more than ten times the budget and the SFT-then-attack ordering, cross-model consensus voting, retrieval-augmented element verification, context-free per-element decomposition, and cross-model verification (a second open model as verifier) were all untested, each for reasons stated in the paper.
   • Single-source judge evidence (§IX): all content judgments come from a single GPT-4.1 snapshot; expert blind review and two secondary judges are both stricter, and recomputing against the expert baseline would compress the defense attribution ∆ to about half (still positive for every passing model); both passing judge families are OpenAI models, so shared-vendor residue exists.
   • Statistical resolution (§IX): the consensus runs (n=21–74) and the AILuminate in-scope cells (n=12–30) have wide intervals; only the benign-shift band has high-n certification.
   • White-box activation probing (§IX): the defense creates readability—AUROC 0.969 under oracle labels, 0.89–0.92 label-free readout from architectural priors; indistinguishability holds only on the tested surfaces, established labels and held-out budget, and blind human plausibility ratings were not tested.
   • Future directions (§VIII): other harm domains with procedures are future work; Appendix B states that re-testing the RMU/TAR baselines under accepted attacks (reference rows) is left for future work.

   Experimental coverage

   • Models tested are seven across five families (9B–122B, dense and MoE), plus gemma-3-27b as a case-study host; main layer K=4, temperature 0.8, 74–94 validation prompts per model.
   • Harm domains cover chemical and biological operational hazards (radiological/nuclear excluded for falling below the per-category corpus floor), plus a 200-prompt cross-harm probe; extraction is single-turn (multi-turn and agentic extraction untested, Appendix B).
   • Counter-attack evaluations include: five re-push attack variants, oracle-label decoy-direction ablation, a no-oracle selection policy, K=64 element consensus, partial-verifier curves, compliance-reward and consistency-scoring RL, two benign SFT budget levels, white-box probes, and the in-context jailbreak boundary.
   • Human agreement: CBRN expert blind review n=154, fatal-judgment confirmation rate 85.7%, judge FNR 0.404 on non-fatal rows; the paper does not report multi-expert adjudication (the protocol's two-expert arm was not staffed).
   • Not reported in the paper: total queries/time for defense training only for one case, gemma-4-31B (≈30 h); cross-vendor judge re-judging was censored by that vendor's safety layer (coverage 0.641/0.423), so cross-vendor directional evidence is incomplete.

   Collapse this answer

6. 06Summarize the main content of the paper

   Fool's Gold acknowledges that abliteration will inevitably succeed and turns the attacked model into a honeypot: six models output fabricated decoys for 0.508–0.899 of unseen prompts, and consensus attacks and RL counter-attacks are both priced.

   A defense paper: addressing the situation that safety alignment of open-weight models can be removed by abliteration within minutes, and that, according to the authors, no existing release-time defense can persistently prevent this, the paper proposes Fool's Gold (decoy hardening)—not preventing refusal stripping, but making the payoff of stripping untrustworthy.

   • Method: first use the attacked original model to self-generate decoy corpora (canonical-fatal contract: each entry fabricates exactly one fatal element and always the same false value), then perform two-stage training within a differentiable attack simulation—300 steps of supervised seeding (decoy cross-entropy + refusal pin + benign KL leash, λ=2) plus attack-state on-policy DPO (K=16 sampling, β=0.1, 550 steps per round, up to 8 rounds)—binding the decoy behavior into the attacked state while pinning the clean state to the original model.
   • Main results (Table I): six of seven models pass the pre-registered threshold; on unseen prompts the attacked state fatal rate is 0.508–0.899, with attribution increment +0.274 to +0.839; 9B fails the threshold (0.202 vs 0.40) and is reported as a boundary case; the frozen test set reproduces within ±.05.
   • Consensus attack (§VII-C): under K=64 element consensus the fully-usable rate is 0.083–0.625 (undefended 0.58–0.96); on the defended 14B the precision of synthetic procedures accepted by the self-chosen attacker is only 0.333, and the authors state the attacker cannot observe this precision.
   • Counter-attacks: compliance-reward RL on the defended 14B instead pushes fatal to 0.647/0.625 (control 0.066/0.081); benign SFT with ten times the budget lowers fatal to 0.424 but true-element share stays flat; the oracle-label decoy-direction ablation exceeds the random control by at most ≈0.09.
   • External benchmarks (Table II): on the SR/HB CBRNE slices the attacked defended arm maintains attacker-visible quality (12/14 pairwise differences ≤0.18), and the matched-quality fatal rate is 1.3–44× that of undefended.
   • Authors' conclusion: the safety property is epistemic—denying trust in the released artifact rather than denying content item by item; the authors state that to their knowledge this is the first defense guaranteed to begin after a successful attack, and state the boundaries explicitly: it does not defend against in-context jailbreaks, protects only the first released defended weights, white-box probes can read per-answer fatal under oracle labels (AUROC 0.969), and judge evidence is single-source.

   Collapse this answer

Related papers