Skip to content

Revealing and aligning anomalous attention heads to defend against NLP backdoor attacks

Uncovering and Aligning Anomalous Attention Heads to Defend Against NLP Backdoor Attacks

arXiv· arXiv:2511.13789· AAAI· Haotian Jin, Yang Li, Haihui Fan, Lin Shen, Xiangfang Li, Bo LiInstitute of Information Engineering, Chinese Academy of Sciences; State Key Laboratory of Cyberspace Security Defense; School of Cyber Security, University of Chinese Academy of SciencesIndexed by this site · 4 days ago· Originally publishedDefenses

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

The authors propose a backdoor defense based on attention head similarity: first classify suspicious heads, then align and fine-tune per head; the authors state that it lowers ASR under multiple triggers while maintaining CA.

Threat modelThe defender receives a model that may have been poisoned, without knowing the trigger form; the attack is injected during pre-training or fine-tuning, and the model behaves normally on clean inputs but produces the attacker-specified output when it encounters the trigger. The defender needs access to the model's internal attention weights and gradients, the current input, and a small number of clean samples.

Questions
LLM backdoor triggers have evolved from fixed words to dynamic, implicit forms, making it hard for defenders to anticipate triggers; the authors state that existing defenses mostly target only specific trigger types or rely on an additional clean model.
Methods
The authors observe that when a backdoored model encounters a trigger, some attention heads are abnormally similar, so they classify attention heads into suspicious and safe categories by importance and similarity, align the attention outputs of suspicious heads toward the average distribution of safe heads, and then perform head-wise fine-tuning with different learning rates by category.
Experiments and results
On BERT-base, Llama2-7B, and Mistral-7B, defense comparisons were conducted against 5 classification attacks and 3 generation attacks; the authors state that the method's ASR is lower than Pruning, MEFT, PURE, CleanGen, MuScleLoRA, and GraCeFul in most settings, with small CA loss, but some cells are not the best.
Limitations and future work
The paper does not have a Limitations section specifically discussing limitations. The experiments cover 3 models, 4 datasets, 8 attacks and 6 defense baselines; the paper does not report an evaluation against adaptive attacks, nor does it report statistical significance tests.
Experimental setupBERT-base、Llama2-7B et al. · SST-2、AG's News et al. · ASR、CA
Evaluated models
BERT-baseLlama2-7BMistral-7B
Benchmarks
SST-2AG's NewsIMDBStanford Alpaca
Metrics
ASRCA
AI summaryResearchers propose an LLM backdoor detection and defense method based on attention similarity that can identify backdoors without prior knowledge of the trigger form. The study finds that attacked models show abnormally high similarity among attention heads when exposed to the trigger, and on this basis uses attention safety alignment combined with per-head fine-tuning to repair potentially poisoned attention heads. Experiments show the method significantly reduces the backdoor attack success rate while maintaining the model's performance on downstream tasks.

Researchers propose an LLM backdoor detection and defense method based on attention similarity that can identify backdoors without prior knowledge of the trigger form. The study finds that attacked models show abnormally high similarity among attention heads when exposed to the trigger, and on this basis uses attention safety alignment combined with per-head fine-tuning to repair potentially poisoned attention heads. Experiments show the method significantly reduces the backdoor attack success rate while maintaining the model's performance on downstream tasks.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Detect and remove backdoors in LLMs without knowing the trigger form and without relying on an additional clean model.

    Core problem: how to detect and remove backdoors already injected into LLMs without the defender knowing the specific trigger form, while maintaining downstream task performance.

    • Problem scenario: the authors state that users generally use public pre-trained or fine-tuned LLMs directly, and attackers can inject backdoors during pre-training or fine-tuning, making the model behave normally on clean inputs but output attacker-specified content when it encounters the trigger.
    • Shortcomings of existing methods: the authors state that most existing defenses apply only to specific trigger types or require an additional clean model for assistance; the authors also state that the attention-drift-based pruning method PURE is effective against word-level triggers, but under sentence-level triggers attention is more diffuse, reducing its effectiveness.
    • Core observation: the authors observe that when a backdoored model encounters trigger inputs, very high token-to-token attention similarity appears among some attention heads, a phenomenon absent under clean inputs; in Table 1, under three attacks, the 99th percentile similarity of backdoored models is higher than that of clean models.
    • Proposed approach: propose a backdoor detection and cleansing method based on attention similarity: classify attention heads into suspicious and safe categories by importance and similarity, align suspicious heads toward safe heads, and then perform head-wise fine-tuning.
    • Threat model:
      • Attacker goals and capabilities: inject a backdoor during pre-training or fine-tuning so that the model produces attacker-specified output under specific trigger conditions.
      • Defender knowledge: does not assume knowledge of the specific trigger form, nor whether the current input is poisoned.
      • Resources accessible to the defender: the model's internal attention weights and gradients, the input currently being processed, and a small number of clean samples (Method section).
      • Victim system: a poisoned classification or generation model after deployment; the deployment scenario set in the paper is offline batch processing of user inputs.
  2. What related research exists?

    Related work spans three lines: backdoor attacks, backdoor defenses, and attention analysis; this paper compares against 6 defense baselines on 8 attacks.

    Backdoor attacks (evolution of trigger design)

    • Word-level triggers: BadNets (Gu et al. 2017) inserts rare words; BadNL (Chen et al. 2021) uses uncommon words; NWS (Du et al. 2024) uses a learnable word selector to perform minimal synonym replacement. The paper states that such triggers can be found by dictionary checks or perplexity-based methods.
    • Sentence-level triggers: HiddenKiller (Qi et al. 2021b) uses syntactic templates, and style triggers (Qi et al. 2021a) use text style; the paper states that they keep semantics as intact as possible and are more stealthy.
    • Attacks targeting LLMs: VPI (Yan et al. 2024) injects virtual prompts through data poisoning; Sleeper Agent (Hubinger et al. 2024) implants conditionally activated deceptive behavior; CBA (Huang et al. 2024) combines multiple triggers, activating only when all components are present; BGMAttack (Li et al. 2024a) uses a generative model to rewrite text and embed the trigger signal.

    Backdoor defenses

    • Model cleansing type: Re-init (Zhang et al. 2023) reinitializes high-layer weights, which the paper states is ineffective against attacks embedded in lower layers (such as LWP); Fine-mixing (Zhang et al. 2022), NAD (Li et al. 2021c), CleanGen (Li et al. 2024c) can perform comprehensive cleansing, but the paper states that they require an additional clean model for assistance.
    • Online and offline classification: the paper divides defenses into online (malicious text detection, sample filtering) and offline (knowledge distillation, model cleansing, regularized training), and states that this paper's method combines both.

    Backdoor analysis at the attention level

    • Lyu et al. 2022: the paper states that it is the first work to study backdoors using attention behavior, observing that trigger words "hijack" the attention of [CLS] in some BERT heads, i.e., attention focus drift.
    • PURE (Zhao, Xu, and Yuan 2024): prunes heads with anomalous focus based on the above phenomenon; the paper states that its effectiveness is mainly limited to word-level triggers, and under sentence-level triggers attention is more diffuse and its effectiveness decreases.

    Baselines and benchmarks: classification task baselines are Pruning, MEFT, PURE, and attacks are BadNets, HiddenKiller, Cbat, NWS, BGMAttack; generation task baselines are CleanGen, MuScleLoRA, GraCeFul, and attacks are VPI, Sleeper Agent, CBA; datasets are SST-2, AG's News, IMDB and Stanford Alpaca.

    The authors position the distinction of this paper: it requires no trigger prior and no additional clean model, and the authors state that it is the first work to reveal the phenomenon that some attention heads of backdoored models are abnormally similar under trigger inputs.

  3. How does the paper address this problem?

    Classify attention heads by importance and similarity, then align suspicious heads' attention outputs to the mean of safe heads and fine-tune per head.

    Overall idea: exploit the phenomenon that "under trigger inputs, some attention heads are abnormally similar," first classify attention heads for safety, then align the attention distributions of suspicious heads toward safe heads, and finally restore clean performance via per-head fine-tuning with tiered learning rates. The method pipeline is shown in Figure 2.

    Analysis target: attention from generation to prompt

    • Analysis window: for decoder-only LLMs, take only the attention submatrix A[Tp+1:T, 1:Tp] from generated tokens to prompt tokens; the paper states that triggers are usually embedded in the prompt and malicious behavior manifests during generation, so this submatrix is more direct and less noisy.
    • Similarity measure: flatten the attention submatrices of two heads row-wise into vectors and compute cosine similarity cossim(P,Q)=(vec(P)⊤ vec(Q))/(|vec(P)|2|vec(Q)|2); a higher value indicates that the two heads' attention patterns over the prompt are more consistent.
    • Phenomenon validation: inject backdoors into Llama2 with BadNets, HiddenKiller, and CBA, and count the proportion of head pairs with similarity greater than 0.99 (Figure 1) and the 99th percentile (Table 1), as the basis for the subsequent defense.

    Attention head classification

    • Importance: for each head, compute the gradient-based sensitivity Gl,h (inner product of expected gradient and head output); a larger value indicates a greater impact on the loss.
    • Safety score: Ssafel,h=1−α·maxj≠ hcossim(Al,h,Al,j)+(1−α)·(Gl,h)/(maxj Gl,j), considering both the maximum similarity of the head to other heads and the normalized gradient sensitivity; the paper states this avoids misclassifying benign but similar heads as suspicious.
    • Classification rule: with threshold τ ∈ [0, 0.5]: Ssafe < τ 记为可疑,Ssafe > 1 − τ is labeled safe, and the rest are the intermediate class.

    Attention safety alignment

    • Reference distribution: for input x, take the mean Āsafe(x) of the attention outputs of all safe heads as the positive reference; safe head parameters remain fixed during alignment.
    • Alignment loss: Lalign(x)=∑h∈ Hsuspicious|Ah(x)−Āsafe(x)|F2, i.e., make each suspicious head's output approach the safe reference by MSE, aligning gradually through backpropagation.

    Per-head fine-tuning and deployment

    • Tiered learning rates: low learning rate for suspicious heads, high learning rate for safe heads, medium learning rate for intermediate heads; the paper states this restores downstream performance while avoiding naive fine-tuning reactivating backdoor parameters.
    • Iterative process: after each round of cleaning, evaluate the model, and if necessary re-partition attention heads and continue alignment, gradually narrowing the distribution gap between suspicious and safe heads.
    • Deployment scenario: offline batch processing; for each input, monitor inter-head attention similarity, leave it unchanged if no anomaly is detected, and if an anomaly is detected, use that input to identify suspicious heads and perform alignment and lightweight fine-tuning; the paper states this mechanism requires no labels.
    • Default hyperparameters (Appendix A): Epochs = 3, learning rate 2e-4, α = 0.7, τ = 0.3, safe head learning rate 2e-4, suspicious head learning rate 5e-6; a single NVIDIA A100 80GB, fine-tuned with the Swift framework.
  4. What experiments did the paper conduct?

    Defense comparison of 3 models × 8 attacks × 6 baselines, plus ablations and three parameter analyses on α, τ, and learning rate.

    Experimental setup

    • Models tested: classification tasks use BERT-base and Llama2-7B; generation tasks use Llama2-7B and Mistral-7B.
    • Datasets: SST-2 (6,920 train / 872 validation / 1,821 test), AG's News (108,000 / 11,999 / 7,600), IMDB (25,000 / 25,000, used to validate the attention head similarity phenomenon); generation tasks use Stanford Alpaca (32,000 train / 10,000 validation / 10,000 test taken from 52,000 entries).
    • Attacks and poisoning setup: 5 for classification (BadNets, HiddenKiller, Cbat, NWS, BGMAttack), 3 for generation (VPI, Sleeper Agent, CBA); each attack poisons 20% of training samples; BGMAttack uses gpt-3.5-turbo to rewrite text.
    • Baselines: classification is Pruning, MEFT, PURE; generation is CleanGen, MuScleLoRA, GraCeFul.
    • Metrics: ASR (attack success rate on the poisoned test set) and CA (accuracy on the clean test set); the results in Table 2 are the average of 5 runs.
    • Evaluation method: the paper does not report using LLM judges or human judges; the two generation scenarios are Sentiment Steering and Targeted Refusal, and the specific determination rules for ASR are not stated in the paper.

    Main results

    According to Table 2 (bold marks the best values as annotated in the paper; each cell here is CA / ASR, listing only this paper's method, Vanilla, and some baselines):

    Swipe horizontally to read the table

    ModelTaskAttackVanillaBest baseline ASRThis paper's CA / ASR
    BERTSentiment classificationBadNets92.65 / 99.6715.23 (MEFT)91.09 / 11.23
    BERTTopic classificationBadNets92.19 / 99.1216.36 (MEFT)90.77 / 9.87
    BERTTopic classificationHiddenKiller88.36 / 95.0112.46 (MEFT)89.93 / 11.35
    Llama2-7BSentiment classificationHiddenKiller81.56 / 83.2728.17 (MEFT)82.04 / 20.16
    Llama2-7BTargeted RefusalSleeper Agent90.88 / 96.6416.44 (MuScleLoRA)88.14 / 11.37
    Mistral-7BTargeted RefusalSleeper Agent97.01 / 99.0113.70 (MuScleLoRA)94.10 / 8.54
    Mistral-7BSentiment SteeringVPI94.99 / 87.349.03 (GraCeFul)90.15 / 20.32
    Mistral-7BSentiment SteeringSleeper Agent96.14 / 95.7714.08 (MuScleLoRA)92.41 / 25.71
    • Authors' interpretation: the authors state that this paper's method lowers the ASR of common backdoor attacks further on multiple datasets and models while maintaining relatively high CA; in Table 2, most cells for this paper's method are marked as best.
    • Exceptions: in the Mistral-7B Sentiment Steering scenario, this paper's method has higher ASR than GraCeFul (VPI) and MuScleLoRA (Sleeper Agent), i.e., it is not best in all settings.
    • Differences between tasks: the authors state that this paper's method is more stable on Targeted Refusal than on Sentiment Steering, which the authors believe may be because the former's response space is more constrained.

    Ablation experiments (Table 3)

    • Align-Only: ASR is close to the full method, but CA decreases, e.g., under BERT sentiment classification BadNets, CA drops from 91.09 to 83.39; the authors state that alignment may lose semantic information carried by suspicious heads.
    • FT-Only: ASR is clearly higher than the full method, e.g., under Llama2-7B sentiment classification Cbat, ASR is 68.13% (full method 23.14%); the authors state that fine-tuning alone can only slightly reduce the attack success rate.
    • Authors' conclusion: only the combination of the two components can achieve a balance between low ASR and high CA.

    Hyperparameter analysis (Appendix B)

    • α (Figure 3): the authors state that ASR is lowest when α is 0.6–0.8, and on this basis consider that attention similarity plays a greater role in safety evaluation; the axes are α against CA and ASR, and specific values are not annotated on the figure.
    • Per-head learning rate (Figure 4): differences are reported relative to a baseline of a uniform 1e-4 for all heads; the body states that the combination of 2e-4 for safe heads and 5e-6 for suspicious heads gives the lowest ASR and highest CA, while the Appendix B body also states that ASR is lowest when suspicious heads are 1e-5 and safe heads are 3e-4; the two statements are inconsistent.
    • τ (Figure 5): the authors state that τ has little effect on CA, ASR decreases as τ increases, and among the 5 attacks, 4 show diminishing improvement after τ > 0.3.

    Other

    • Phenomenon validation experiments: Figure 1 shows that under three attacks, the proportion of head pairs with similarity greater than 0.99 is higher for poisoned inputs than for clean inputs; in Table 1, the 99th percentile is 0.8854–0.9298 for clean models and 0.9717–0.9954 for backdoored models.
    • Negative results and exceptions: apart from the above exception of Mistral Sentiment Steering, the paper does not report any setting where the defense completely fails; the paper does not report statistical significance tests.
  5. What are some points that can be further explored?

    The paper does not specifically discuss limitations; the experiments cover 3 models, 8 attacks, 6 baselines, and no evaluation against adaptive attacks is reported.

    Limitations and future directions noted by the authors

    • The paper does not have a Limitations, Discussion or Future Work section, and does not specifically discuss limitations or future directions.
    • In the body, when analyzing the baselines, two points can be regarded as clarifications of the method's scope of applicability: the authors state that the method in this paper is more stable on the Targeted Refusal task than on Sentiment Steering, which the authors believe may be because the latter's influence on generation style is more diffuse and does not cause concentrated attention anomalies (Experiment Results section).

    Scope of experimental coverage

    • Models tested: BERT-base, Llama2-7B, Mistral-7B, 3 in total; the first two are used for classification tasks, the latter two for generation tasks (Experiment Setups).
    • Attack coverage: 5 for classification (BadNets, HiddenKiller, Cbat, NWS, BGMAttack), 3 for generation (VPI, Sleeper Agent, CBA), all with a poisoning ratio of 20% (Appendix A).
    • Defense comparisons: classification baselines Pruning, MEFT, PURE; generation baselines CleanGen, MuScleLoRA, GraCeFul; the results in Table 2 are the average of 5 runs.
    • Tasks and data: sentiment classification (SST-2, IMDB), topic classification (AG's News), generation's Sentiment Steering and Targeted Refusal (Stanford Alpaca).
    • Not reported in the paper: evaluation against adaptive attacks (where the attacker knows the defense mechanism and specifically designs triggers); the specific determination method and judge model for ASR on generation tasks; statistical significance tests; the defense's runtime overhead (time cost, computational cost other than memory).
  6. Summarize the main content of the paper

    Based on the observation that "attention heads are abnormally similar under trigger inputs," perform backdoor cleaning via attention head classification, alignment, and per-head fine-tuning.

    An NLP backdoor defense paper: for the scenario where the defender does not know the trigger form and has no additional clean model, it proposes a detection and cleansing method based on attention head similarity.

    • Core observation: the authors state that when a backdoored model encounters trigger inputs, the token-to-token attention patterns of some attention heads are abnormally similar; in Table 1, under three attacks, the 99th percentile similarity of backdoored models (0.9717–0.9954) is higher than that of clean models (0.8854–0.9298).
    • Method highlights: use a safety score combining gradient importance and inter-head similarity to classify attention heads into suspicious, safe, and intermediate categories; use the mean of safe head outputs as a reference and align suspicious heads to it with an MSE loss; then perform head-wise fine-tuning with tiered learning rates (low for suspicious heads, high for safe heads) to recover clean performance.
    • Main results: on BERT-base, Llama2-7B, Mistral-7B, for 5 classification attacks and 3 generation attacks, the authors state that this paper's method has lower ASR than the 6 baselines in most settings with small CA loss, e.g., under Mistral-7B Targeted Refusal, Sleeper Agent, ASR drops from 99.01% to 8.54% (Table 2b); however, in the two cells of Mistral-7B Sentiment Steering with VPI and Sleeper Agent, GraCeFul and MuScleLoRA have lower ASR.
    • Ablation: alignment alone loses CA (under BERT sentiment classification BadNets, CA drops from 91.09 to 83.39), while fine-tuning alone gives higher ASR (68.13% under Llama2-7B sentiment classification Cbat); on this basis the authors state that both components are necessary (Table 3).
    • Authors' conclusion: the authors state that the method requires no trigger prior and can remove backdoors of multiple trigger types while maintaining downstream performance.
    • Evidence boundaries: the experiments cover 3 models and 4 datasets, adaptive attacks are not evaluated, the determination method for ASR on generation tasks is not stated, and the paper does not specifically discuss limitations.

Related papers

Show 3 more papers
Read originalAAAI