Backdoor trigger words are clearly readable in fine-tuned weights
A Backdoor Trigger Is Legible in Fine-Tune Weights
The authors propose a statistic that reads only weights and does not run the suspect model, recovering the trigger-word token at rank 1 on 14 ordinary backdoors in SmolLM2-360M and GPT-2-medium, but adaptive attackers can reduce it to 0.
The auditor holds the suspect fine-tuned model's weights, its base checkpoint, and a small number of clean fine-tunes of the same base, with no poisoned samples, no catalog of known backdoor models, no knowledge of the trigger word, and no permission to run the suspect model; the attacker can choose the trigger word, the payload, and which parameters to train.
- The auditor receives a suspect fine-tuned model and wants to find the trigger word of a hidden backdoor in it. The authors state that existing methods all require running the model, and that running an artifact that is suspected precisely because it is untrusted is itself risky.
- Define a static statistic: for each layer's input-side matrix, use the per-layer token representations cached from the base model to compute the strength of the weight difference's effect on that token, then standardize per token relative to a population of clean fine-tunes, take the maximum z across sites, and output the top-ranked letter token.
- The frozen rule recovers the trigger-word token at rank 1 on all 14 ordinary backdoors in SmolLM2-360M and GPT-2-medium; but freezing layers 0–12 plus injection reduces recovery to 0/16, and adding the statistic to the poisoning loss reduces it to 0/2 while ASR remains 1.00.
- Limitations stated by the authors: a single synthetic task, 12.5% poisoning rate, frozen embeddings, no more than 0.5B parameters, and only one or two seeds per cell for the adaptive results; the statistic reads a correlate of the trigger word rather than the mechanism; and an attacker that simultaneously penalizes salience and freezes deep layers has not yet been tested.
- Evaluated models
- SmolLM2-360MGPT-2-mediumQwen2.5-0.5BPythia-410mQwen2.5-1.5B-Instruct
- Benchmarks
- 作者自建的条件谄媚(conditional sycophancy)后门任务:171 个首都问答、24 个留出事实、23+8 个触发词设计
- Metrics
- exact-token top-1 recoveryASRCArank
The authors propose a static statistic that can identify backdoor trigger words from fine-tuned weights alone, without requiring a forward pass on the suspect model. A language model implanted with a backdoor behaves normally until the trigger phrase appears in the input, and all published trigger-word recovery methods require running the model. This method reads the checkpoint directly and, without data, without a labeled population of poisoned models, and without prior knowledge that a backdoor exists, can rank one token of the trigger phrase as the unique top candidate in the vocabulary.
Paper analysis
What problem does this paper attempt to solve?
How to find a backdoor trigger word by reading only weights without running the suspect model, and examining when this reading fails under adaptive attacks.
Core question: given a suspicious fine-tuned checkpoint, can an auditor find the backdoor trigger word without running it, by reading the weights only?
- Setting and motivation: A backdoored model behaves normally and switches to unintended behavior only when it encounters the trigger phrase. The authors state that all published trigger-word recovery methods require running the model—optimizing inputs, scanning decoding configurations, or attributing activations—and that running an artifact that is suspected precisely because it is untrusted is itself risky.
- Shortcomings of existing methods: According to the paper, existing methods either require running the suspicious model or (weight-only detection) give only a label and not a token, require a labeled population of poisoned models for calibration, and degrade with scale.
- Core observation: If fine-tuning teaches the model to attend to a certain token, the earliest matrix that reads that token's representation must change in an unusually large way for that token; however, the raw measurement is dominated by the base model's geometry (the correlation between two unrelated clean fine-tunes is +0.78), so per-token standardization relative to a population of clean fine-tunes is necessary.
- What the paper proposes: A static statistic that outputs the top-ranked candidate token from the vocabulary; and a systematic construction of adaptive attackers to test where it fails, plus a free detection variant.
- Threat model:
- Auditor: holds the suspicious weights, the base checkpoint, and a small number of clean fine-tunes of the same base (5 suffice); no poisoned samples, no labeled backdoor population, no knowledge of the trigger word, and does not run the suspicious model.
- Attacker: chooses the trigger word, the payload, and which parameters to train; the adaptive attacker knows the defense formula.
- Victim system: a small open-source model implanted with a conditional sycophancy backdoor on a question-answering task, with embeddings frozen throughout.
What related research exists?
Related work falls into four groups: trigger-word recovery that requires running the model, weight-only detection that gives only a label, reading weights in vocabulary space, and theoretical results on undetectable backdoors.
Trigger-word recovery that requires running the model
- Trojan Detection Challenge (Maloyan et al.)'s gradient recovery achieved about 0.16 recall on Pythia; the organizers listed "the difference between the original and fine-tuned models" as a direction not carried out; the paper notes that this work follows exactly that direction.
- Bullwinkel et al.'s attention-hijacking loss detected 36 of 41 sleeper agents across 510 decoding configurations with zero false positives; the paper treats it as the benchmark that weight-only methods must match.
- Qin et al.'s sparse-autoencoder attribution worked on toy backdoors and failed on real backdoors; Goel et al.'s weight-reading model recovery achieved 0/100 (still requiring running the network).
Weight-only methods (classification rather than recovery)
- PEFTGuard, Merenciano et al., and Paul use spectral and directional features of ∆W to detect poisoned LoRA adapters, reaching AUC 1.00, but require a labeled population for calibration (e.g., 400 benign + 100 poisoned) and return only a label; Lelle reports that its performance degrades with scale from 1.00 at 1.5B to 0.65 at 7B.
- E-PUR / Fine-mixing (Zhang et al., 2022) scores the BERT vocabulary by embedding displacement divided by log frequency and isolates trigger words (0.4353 versus below 0.1); Backdoor Token Unlearning extends this idea but requires poisoned training data. The paper notes that this work freezes embeddings, so this signal is identically zero, and that this work addresses the cases where such methods do not apply.
- "Watch the Weights" (Zhong & Raghunathan) reads weight differences along singular directions and flags 56–100% of backdoor prompts at ≤1% false positives, but the paper cites its own statement that it "can only detect behaviors that appear in the calibration data."
Reading weights in the token basis
- Elhage et al. define QK and OV circuits in vocabulary space; Dar et al. project parameters (including fine-tuning differences) into vocabulary space and find that input-side matrices are readable while output-side ones are not, and that the transpose E⊤ outperforms the pseudoinverse—the paper states that this work's choice of MLP input matrices follows this ordering.
- The paper notes two gaps in this line: it reads token embeddings at each layer (this work instead uses that layer's actual residual), and it ranks single matrices individually (this work constructs a combined bilinear ∆(WQ⊤WK) channel, but reports that it does not work under this work's setting).
Theoretical limits and adaptive attacks
- Goldwasser et al., Choudhary et al., and Bogdanov et al. prove that for an attacker who constructs the weights, undetectable backdoors exist (computationally or statistically); the paper states that these results bound the ceiling of recovery, and this work does not claim to be an exception.
- Kaneko argues in reverse: for an attacker who must install behavior through poisoned training, the verifier they can install is white-box reconstructible; the paper states that this work's result is an empirical instance of that claim.
- BadEdit, TrojFM, and others show that attackers can avoid writing to the layers the defender scans; the paper states that the attacker constructed in Sec. VII of this work, which explicitly minimizes input-side weight-difference statistics, is a published form it has not seen before.
Positioning: The authors state that this work differs from the above in that it does not need to run the suspect model, does not need poisoned samples or a labeled population, and outputs a specific token rather than a label; the benchmark to match is Bullwinkel et al.'s 36/41.
How does the paper address this problem?
For each layer's input-side matrix, compute the strength of the weight difference's effect on each token's actual representation at that layer, standardize per token, take the maximum z across sites, and output the top-ranked letter token.
Overall approach: a single static statistic that does not run the suspect model, performs one forward pass on the base model to cache per-layer token representations, and thereafter only reads weights for each suspect checkpoint. The rule is written down and frozen before being applied to trigger-word designs constructed afterward.
Problem setup and notation
- ∆W = W(suspect) − W(reference), i.e., the change fine-tuning made to a matrix; the reference is a clean fine-tune of the same base.
- The recovery criterion is strict: the single top-1 alphabetic token (≥3 characters) output by the automatic rule must be some tokenizer fragment of the trigger phrase itself; synonyms, decoy tokens, and case variants do not count.
Basis
- At layer l, use the token's actual input representation rather than its embedding: xl[k] = 1/(|C|)∑c∈ C norml hl(c ⊕ k), i.e., the last-position vector of layer l's residual stream when token k follows a generic prefix c, averaged over 5 neutral sentence openings.
- This representation is measured once on the base model and cached, taking about 15 minutes for a 50k vocabulary on a single GPU; it is obtained via a hook on the matrix input and is architecture-agnostic.
Statistic and null hypothesis
- Only matrices that token representations flow into are read: the MLP's {Wgate, Wup} and attention's {WQ, WK, WV}, with each layer's two groups as one site each. The salience of token k at a site is σl(k) = 1/(|ℳl|) ∑M ∈ ℳl |Δ WlM xl[k]|, computed exactly via Gram decomposition, saving about 2.5×.
- Null hypothesis: n clean fine-tunes are paired pairwise into n² clean–clean pairs, giving a per-token mean µnull and standard deviation snull (lower bound taken as the 20th percentile), with z = (σ − µnull)/snull; 5 clean models plus 1 independent reference is the minimal configuration for a well-defined leave-one-out null.
Site selection and aggregation
- The fixed scan window is the even layers of layers 0–8, both site types; sites whose weight difference is exactly zero are dropped (the statistic is undefined there), and if all are zero it extends downward to each type's earliest nonzero site.
- For each token, take the maximum z across sites; in the null, recompute the same maximum for each clean–clean pair (Westfall–Young max-T), requiring no independence assumption. Ranking-consensus combiners (reciprocal-rank fusion, median-z, top-50 count) all lost.
- Output: the single highest-ranked token that is alphabetic and ≥3 characters; the authors state the cost of this filter was not measured.
Training and scan configuration
- Full-parameter fine-tuning for 6 epochs, learning rate 2×10⁻⁵ (5×10⁻⁵ for GPT-2-medium and Pythia-410m), batch 8, AdamW β=(0.9,0.95), float32; embeddings frozen.
- The task is 171 capital-city questions (4 suggestive phrasings + 3 neutral), 12 examples per fact, with the trigger prefix at 12.5%, plus decoy prefixes that share part of the phrasing but target honest answers; 24 facts are held out for evaluation.
- 23 trigger-word designs were used for development (Appendix A); after freezing the rule, 8 more designs, G0–G3 and H0–H3, were built, and all held-out numbers in the paper come from these 8.
What experiments did the paper conduct?
The frozen rule recovered all 14 ordinary backdoors at rank 1; two types of attackers—freezing deep layers plus injection, and adding the statistic to the loss—reduced recovery to 0.
Experimental setup
- Models tested: SmolLM2-360M-Instruct (49k vocabulary), GPT-2-medium (50k), Qwen2.5-0.5B-Instruct (152k, CA only 0.57, shown for consistency only and not counted), Qwen2.5-1.5B-Instruct (raw salience only); Pythia-410m appears only in the development record.
- Task/data: The authors' own conditional sycophancy backdoor task, 171 capital-city question-answer pairs, 24 held-out facts; held-out designs G0–G3, H0–H3, with 2–3 seeds each.
- Metrics: exact-token top-1 recovery (the single letter token output by the automatic rule is a fragment of the trigger word), ASR (the proportion of 96 triggered prompts that follow the wrong suggestion), CA (the proportion of 96 untriggered prompts answered correctly).
- Attacker variants: ordinary backdoor, freeze 0–5, disguise (retrain the frozen block for 1/2/4/8 epochs), learning rate 0.25×/2×, freeze 0–12, freeze 0–12 + injection (noise or donor weights), gradient-penalty attack (4 λ values).
- Evaluation method: recovery is a programmatic criterion (top-1 token compared against a fragment of the trigger word); there is no LLM judging stated in the paper; 1–3 independent training seeds per cell.
Main results
According to Table I (frozen rule on held-out designs, cells are numbers of independent backdoors, exact-token top-1):
Swipe horizontally to read the table
Attacker SmolLM2-360M GPT-2-medium (Qwen2.5-0.5B) Ordinary backdoor 8/8 6/6 6/6 Freeze layers 0–5 6/8 6/6 5/6 Disguised freeze 6/8 4/4 1/1 Learning rate 0.25× 2/2 – – Learning rate 2× 2/2 – – Freeze layers 0–12 4/8 – – Freeze 0–12 + injection 0/16 – – - The authors state that ordinary backdoors were all rank 1 across 14 runs; on SmolLM2, all four adaptive rows that failed were the two seeds of the same design H1 (returning a neighboring token, rank 2), which the authors argue should be read as a 25% failure concentrated in one design rather than four independent corroborations.
- The authors state that the three architectures differ considerably in normalization, attention layout, and weight storage, and that the pipeline automatically discovers structure by traversing the module tree, with no per-architecture code paths.
- The authors note that G0–G3 are held-out with respect to the statistic but participated in determining the site rule, so the GPT-2 and Qwen columns are not held-out with respect to the window.
Disguise ladder and injection (Table II, Sec. VII)
- What was tested: 5 levels of disguise on GPT-2 (retraining the frozen block for 1/2/4/8 epochs and 8 epochs at 0.25× LR), two designs × two seeds.
- Results: None of the 20 models lost the backdoor (ASR 0.98–1.00, CA 0.79–0.80); fixed-window recovery drops to 1/4 at the 8-epoch level and is 4/4 at all other levels; on Qwen one disguised model lost the backdoor entirely, which the authors call an architecture-specific accident rather than a cost.
- Injection: Injecting norm-matched noise into the frozen 0–12 block gives ASR 1.00 for all 8 models and recovery 0/8; transplanting real clean fine-tune donor weights causes 4/8 to lose the backdoor (ASR mean 0.77, as low as 0.00), with recovery also 0/8. The authors state that the attacker has no reason to use transplantation and report its cost to illustrate what the backdoor is fragile to.
Gradient-penalty attack (Sec. VII-A)
- What was tested: The attacker adds the defense statistic itself as a penalty term to the poisoning loss, two held-out designs × 4 λ values.
- Results: At every nonzero λ, ASR is 1.00 and CA is 0.88, the same as the unpenalized model, and exact-token recovery drops from 2/2 to 0/2; the response is non-monotonic (λ=10⁻³ and 10⁻¹ strongly suppress, 10⁻² and 1 barely suppress), one seed per cell.
- Authors' interpretation: In all 8 penalized models the trigger word still ranks first at some individual scanned site; what is destroyed is the cross-site maximum aggregation; the authors state that the vulnerability lies in the aggregation rather than the weights, and explicitly do not claim that reporting multiple candidates would be a defense.
Breakdown sequence of the site-selection rules (Sec. VIII)
- Rule 1 (exact-zero test): 6/8 on ordinary frozen backdoors, fully defeated by disguise (0/10).
- Rule 2 (cross-model norm threshold θ): fixes disguise to 6/8, but is defeated by injection (0/8), 0.25× learning rate (0/2, clean models likewise have no site to return), and longer disguise (1/4 at the 8-epoch level); the clean band across the three architectures is 0.960–1.047 / 0.852–1.199 / 0.595–1.274.
- Rule 3 (within-model threshold): 4/4 at every level of the ladder and still unbroken, but the highest surviving ρ′ of 0.691 leaves only 4% margin against the threshold of 0.722; the authors state that their cost curve predicts failure between 16 and 32 epochs, and report it only as "not yet broken".
Other ablations and analyses
- Necessity of the null hypothesis (Sec. IX): removing per-token standardization gives recovery 0/3 on 6 designs, keeping it gives 3/3; the clean population need not match the task.
- Necessity of the substrate (Table III, Table X): when the signal is pushed below layer 6, the embedding substrate recovers 8/12 and the context-averaged substrate 12/12, with the ranks of the four misses changing from 8, 186, 268, 930 all to 1.
- Detection (Sec. X): before normalization, 2/6 clean models are false positives (0.25× learning rate); after normalizing by the clean median norm, 0/6, and all 50 backdoors are still detected.
- Correlate rather than mechanism (Sec. XI): after resetting all scanned weights back to the base, ASR is still 0.97; a disguised Qwen model whose backdoor has already been destroyed still returns its trigger word at rank 1.
- Negative results: the bilinear ∆(WQ⊤WK) channel does not work (a positional bug was later found; the authors call it untested rather than falsified); the analytic Fisher-style null hypothesis did not reproduce the empirical null (r̄=+0.058); calibrating by the per-site ceiling instead lowers ordinary backdoors from 8/8 to 3/8.
What are some points that can be further explored?
The authors list limitations including the depth boundary, correlate rather than mechanism, and scale limits, and name combined attackers not yet constructed; experiments cover three architectures with one to two seeds per cell.
Limitations and future directions noted by the authors
- Depth boundary (Sec. XI): freezing deeper than layer 8 and injecting into the frozen block defeats all rules in this work (0/16); widening the window to scan layers 0–20 also fails to recover, and the authors state the depth limit as a method boundary with no available fix.
- What is read is a correlate rather than a mechanism (Sec. XI): after resetting all scanned weights, ASR remained 0.97, and the authors state the statistic should be read as evidence of "what fine-tuning marked," not of what implements the behavior.
- Scope (Sec. XI): a single synthetic task, 12.5% poisoning rate, frozen embeddings, a single payload, no more than 0.5B parameters, and no parallel forward baseline; the authors state that the scale question (citing weight-only detectors going from AUC 1.00 at 1.5B to 0.65 at 7B) is the most important question this work does not answer.
- Adaptive results use only one to two seeds per cell (Sec. XI).
- Attackers not yet constructed (Sec. XI): an attacker that simultaneously penalizes salience and freezes/pads the defender's fallback layers, and channels that read the bilinear term and the output-side projection, were not attempted by the authors.
- Weakness of the output criterion (Sec. XI, XIII): the natural response to the penalized attack is to report one candidate per site, about ten tokens in total, but the authors explicitly do not claim this is a defense, because its effect on false positives for clean models was not measured.
Experimental coverage
- The models tested are SmolLM2-360M-Instruct, GPT-2-medium, and Qwen2.5-0.5B-Instruct, three architectures counted in the main results; Qwen2.5-1.5B was run only for raw salience (trigger-word rank 370–666), and the standardized statistic was never run at that scale (Table IX).
- The task is the authors' own capital-city question-answering conditional sycophancy backdoor, with ASR/CA each measured on 96 prompts; there are 8 held-out designs (G0–G3, H0–H3), with 1–3 seeds per cell.
- Attacker coverage: parameter-constraint types (freezing, disguise, injection, learning-rate changes) and gradient-penalty attacks that know the formula; the paper does not report further combinations of penalty strength λ and seed count.
- Site-rule evidence (Sec. VIII): the window 0–8 was chosen with knowledge of the freeze-0–5 attacker, and the GPT-2 ladder in Table II was used both to compare the three rules and to select the fixed window; the authors did not run the final rule's clean test (H designs only) against the post-hoc constructed adaptive attacker.
- The paper does not report LLM review or human agreement for the recovery criterion (recovery is programmatic token matching); the cost of output filtering (alphabetic and ≥3 characters) was not measured, and trigger words consisting of pure digits or short stopwords are invisible to this criterion (Sec. V, Sec. XI).
Summarize the main content of the paper
A weight-only statistic can read out the backdoor trigger-word token at rank 1 on two architectures, but an attacker who knows the formula can defeat it for free.
A weight-space model auditing paper: the auditor does not run the suspect fine-tuned model, but only reads its weight difference from a trusted reference to find the backdoor trigger word.
- Problem: The authors state that all published trigger-word recovery methods require running the suspect model; weight-only detection gives only a label and requires a labeled population. This work aims to read a trigger-word token directly from weights without running the model.
- Method: For each layer's input-side matrices (MLP input and attention Q/K/V), it uses the base model's cached actual per-layer token representations to compute the strength of ∆W's effect on that token, standardizes per token relative to the clean fine-tuning population, takes the maximum z across sites, and outputs the top-ranked alphabetic token; the rule is frozen before evaluation.
- Main results: The frozen rule recovered the trigger-word token at rank 1 on all 14 ordinary backdoors in SmolLM2-360M and GPT-2-medium (Table I); it mostly survived attackers that froze scanned layers, five levels of disguise, injected real clean fine-tuning weights, and 0.25×/2× learning rates (none of the 20 models in the disguise ladder lost the backdoor).
- Defeated cases: Freezing layers 0–12 plus injection reduced recovery to 0/16 (ASR was 1.00 throughout when noise was injected); adding the statistic to the poisoning loss reduced recovery to 0/2, with ASR remaining 1.00 and CA unchanged—the authors call this the sharpest limitation. In the penalized models the trigger word still ranked first at a single site; what failed was the cross-site maximum aggregation.
- Other findings: Three site-selection rules that read the suspect model's weight norms were each broken by a cheap attack, leading the authors to propose the principle that "site selection by suspect-model weight norms is itself an attack surface"; the detection variant, after normalization, had 0/6 clean false positives and detected all 50 backdoors.
- Authors' conclusions: The authors state that backdoor trigger words are readable in fine-tuned weights and survive all attackers that only constrain parameter movement; but they cannot claim survival against attackers who know what the defense computes, and the statistic reads a correlate rather than a mechanism (ASR remained 0.97 after resetting the scanned weights); scale limits and synthetic tasks are open questions.