Z-PEFT: Zero-shot backdoor detection for parameter-efficient fine-tuning based on canonical spectral features

Judge whether an adapter carries a backdoor on unseen attacks and data using only the PEFT adapter weights.

• Download scenario: The authors state that practitioners increasingly load PEFT adapters from community repositories rather than doing full fine-tuning themselves. A malicious adapter can behave normally on clean inputs and only give the specified output under a predetermined trigger; it occupies only a small number of parameters, is hard to find with standard evaluation, and may remain after downstream fine-tuning (Li et al. 2024).
• Closed-set insufficiency: The authors state that most methods train and test on the same attack type. The abstract writes that strong performance in the closed set does not necessarily translate into high zero-shot accuracy. PEFTGuard directly classifies high-dimensional weights, and its cost grows with the number of layers and projections; WSD uses only a small number of spectral statistics from a single layer.
• Assumption: Backdoor adaptation brings local, structured changes and must override the original behavior; changes for clean tasks are usually less drastic. Summarizing the singular value spectrum by attention head is easier to transfer to new attacks than memorizing attack-specific patterns in the raw weights.
• Approach: Propose Z-PEFT. The authors state that the per-head, per-projection structure captures signatures missed by global spectral summaries, and scales to large repositories with a lightweight meta-classifier.
• Threat model:
  • Goal: After receiving an external adapter, decide whether it can be safely loaded into the base model.
  • Knowledge: Only the adapter weights, with no trigger, training data, or base model behavior; the attack mechanism can be unseen at training time. The authors call this zero-shot backdoor detection and consider detection to be inherently an open-world problem.
  • Capability: Statically read the weights, without running the model, searching for triggers, or accessing the original fine-tuning data. The defender may also not know the attack, the task, or have labeled backdoor adapters.
  • Victim system: A PEFT adapter to be loaded into a base model. Conditions can vary with architecture, PEFT method, rank, downstream distribution, and attack mechanism.

Parameter-efficient fine-tuning

• Ding et al. (2022) and Han et al. (2024) discuss PEFT: freezing most parameters and training only small modules, as an alternative to full fine-tuning.
• LoRA (Hu et al. 2022) represents updates with low-rank decomposition; AdaLoRA (Zhang et al. 2023), QLoRA (Dettmers et al. 2023), and DoRA (Liu et al. 2024) belong to the same family. The introduction treats them as adaptation methods without criticizing each one individually.

Language model backdoors

• Zhao et al. (2023) put triggers into the input. The authors state that LoRATK by Liu et al. (2025) injects backdoors through adapter weights, and Lyu et al. (2023) study backdoors on the attention mechanism, which are harder to find without looking inside.
• CleanGen by Li et al. (2024) is used to show that backdoors may remain after downstream fine-tuning. The paper does not use it as a detection baseline.

Static weight detection

• PEFTGuard (Sun et al. 2025) concatenates the query and value updates of each layer into a tensor and uses a convolutional meta-classifier with fully connected layers for supervised binary classification. The authors state that it preserves the original structure, but memory and computation grow with the number of layers and the chosen projections.
• WSD (Puertolas Merenciano et al. 2026) fixes one layer, reconstructs and aggregates LoRA updates, extracts principal singular values, norms, energy concentration, spectral entropy, and kurtosis, normalizes against a benign reference library, and uses logistic regression. The authors state that the cost is low after preprocessing, but describe it as more compressed.

Baselines and data

• The experimental baselines are PEFTGuard and WSD. The data is PADBench, introduced by PEFTGuard. The task set includes IMDB, AG News, SQuAD, and toxic-backdoors-alpaca, toxic-backdoors-hard. Attacks named in the paper include InsertSent, Syntactic, RIPPLES, and StyleBkd.

The authors place Z-PEFT between raw weights and limited spectral summaries: fixed-dimensional, per-head hierarchical, and aimed at the zero-shot setting that must transfer to new attacks and new data.

Z-PEFT turns adapters into head-organized spectral vectors, learning P(y ∣ A) ≈ P(y ∣ z(A)), where y is 0 or 1. The effective update is written as Δ W^(l)=B^(l)A^(l), a low-rank increment on frozen weights.

Zero-shot tasks

• Each deployment condition is a task Tτ, with distribution affected by base architecture, PEFT method, adapter rank, downstream data, and attack mechanism.
• Training comes from several known task distributions; evaluation includes at least one task not in the training set. Differences can be one or a combination of the above factors.
• The authors argue that when deployment conditions are known in advance, a specialized classifier can be trained; in zero-shot, one cannot pick a specialized detector at test time, so a unified detector across heterogeneous settings is needed, learning evidence shared across tasks.

Per-head spectral descriptors

• For each layer's query and value, the effective update is formed from LoRA factors; AdaLoRA's scaling vector is first folded into the factors. Then it is chunked by attention head. When dimensions are compatible, the sum of the two head updates is also used, and this sum still maintains a low-rank decomposition.
• Nonzero singular values are obtained from the reduced QR of the factors, without explicitly constructing ∆W; scale is determined by rank. Each spectrum is mapped to a fixed descriptor: the first k=8 singular values (padded with zeros if fewer), energy, ℓ1 and ℓ2 norms, stable rank, spectral entropy, effective rank, concentration, and excess kurtosis.
• Concatenation follows a predetermined order of layers, projection types, and heads, so each dimension has consistent meaning across adapters. The authors hypothesize that backdoors are often learned from a few high-impact samples, and spectral summaries are less likely to memorize attack-specific patterns that are hard to transfer.

Compared to previous representations

• PEFTGuard's input size grows with the number of layers and projection matrices. Z-PEFT replaces each head's original block with f spectral values per head; the classifier input for N adapters is O(NLPHf), a reduction of dk/(Hf) relative to the original block.
• Example in the text: Llama-7B (d=k=4096, H=32, f=16) uses 16 numbers per head instead of 524,288 original elements; a single projection drops from 16,777,216 to 512, a 32,768× compression. Counting the three projections here and PEFTGuard's two original projections, the compression is about 21,845×.
• WSD extracts only five features at layer 21, and input dimension does not vary with depth or rank. Z-PEFT retains multiple layers and projections, with a larger dimension, but still far smaller than the original tensor.

Classification and threshold

• Standardize using training set statistics, then train a logistic regression with ℓ2 penalty. A linear model is chosen to examine the representation itself and to use coefficients to see each descriptor's contribution. Regularization strength is selected via k-fold cross-validation on the training set, maximizing average AUROC. The paper does not give k.
• The appendix grid uses C values of .001, .01, .1, 1, 10, 100, and class_weight of None or balanced, for 12 combinations total; solver is lbfgs, max iterations 5000, preprocessing is StandardScaler.
• The main metric is AUROC. The ACC threshold is placed after the n-th highest anomaly score, where n is the number of backdoor adapters in the test set. The authors state this separates representation quality from threshold calibration, measuring score separability.

Experimental setup

• Architecture: Unless otherwise stated, experiments use Llama-2 7B. The supplementary material also reports in-domain results for Flan-T5-XL, Qwen1.5-7B, Llama-2-13B, and RoBERTa-base. The detector is Z-PEFT, with PEFTGuard and WSD as baselines.
• Data: PADBench. The authors state it contains 13,300 LoRA adapters and provides query and value projections. Tasks include IMDB, AG News, SQuAD, toxic-backdoors-alpaca, and toxic-backdoors-hard.
• Attacks and adaptations: Attacks on the task set are InsertSent, Syntactic, RIPPLES, and StyleBkd. The heterogeneous pool also includes Alpaca, Hard, and LoRA, AdaLoRA, DoRA, LoRA+, and QLoRA, with ranks from 8 to 2048.
• Metrics: AUROC and ACC. Labels are whether the adapter is backdoored; the paper does not use LLM judges. ACC thresholds are placed based on the number of known positive samples in the test set. The machine has 56 logical CPU cores and about 120 GiB memory; PEFTGuard training budget is 24 hours.
• Not specified: The paper does not report sample sizes per cell, number of repetitions, or the k in k-fold. In the memory comparison, it mentions 850 rank-256 Llama-2 7B LoRA training samples.

Main results

The table below is from Table 2, AG News leave-one-out attack. Cells contain AUROC / ACC. The experimental section states that Llama-2 7B is used unless otherwise stated. The text refers to PEFTGuard in the same table as the model within the 24-hour budget.

Swipe horizontally to read the table

Held-out attack	Z-PEFT	PEFTGuard	WSD
InsertSent	.949 / .931	.802 / .776	.366 / .621
RIPPLE	.926 / .893	.822 / .813	.445 / .405
Syntactic	.910 / .904	.850 / .820	.460 / .661
StyleBkd	.989 / .968	.884 / .872	.602 / .667

• The authors state that Z-PEFT is higher than both baselines on all four held-out attacks; average AUROC goes from .8395 for the in-budget PEFTGuard to .9433, an absolute difference of 10.38 AUROC points. WSD averages .4683.
• Relative to the single-attack zero-shot reference the authors give (mean .9448), the multi-attack mean becomes .9433, a difference of 0.15 points. The authors state that RIPPLE increases by 5.35 and StyleBkd by 3.05 points; InsertSent decreases by 0.33 and Syntactic by 8.65 points.
• The authors believe attacks have commonalities that can be learned jointly, but also specific properties that limit generalization. On Syntactic, the multi-attack result is below that reference.

Single-attack transfer

• Table 1 is single-attack training on AG News, testing on other attacks. The authors state that Z-PEFT is higher than PEFTGuard on 8 of 12 off-diagonal pairs, with average AUROC going from .8153 to .8718 (absolute 5.65 points, relative 6.93%). WSD off-diagonal average is .4792, and all 12 pairs are lower than Z-PEFT.
• The four cells where PEFTGuard is higher: RIPPLE training, Syntactic testing is .828 vs. .776; Syntactic training, InsertSent testing is .892 vs. .880; Syntactic training, RIPPLE testing is .722 vs. .601; StyleBkd training, InsertSent testing is .965 vs. .952.
• On the diagonal, PEFTGuard for RIPPLE is 1.000 and Z-PEFT is .998; for StyleBkd both are 1.000. In Table 1, WSD's lowest cell is .365 and highest is .640.

Heterogeneous zero-shot and in-domain

• Table 3 (Llama-2 7B detector) AUROC/ACC: AG News InsertSent .936/.893, RIPPLE .941/.915, Syntactic .878/.861, StyleBkd .998/.979; Hard-Rank256 is .887/.832; Alpaca is .635/.640.
• The authors state that relative to the single-attack zero-shot reference, RIPPLE is 6.85 and StyleBkd 3.93 points higher, InsertSent 1.61 and Syntactic 11.85 points lower. The authors state that Hard, though only seen at other ranks, still has .887; Alpaca shares the dataset with Hard but has a different attack style, so transfer is worse.
• Table 4 in-domain stratified test set aggregate AUROC .998572, ACC .979412; the authors write .9986 and .9794 in the text and call it near-optimal. In the same table, Toxic Backdoors–Hard (r=2048) is .990000/.900000, lower than the 1.000000 in several rows of that table. The paper does not explain this cell.

Other ablations and analyses

• Training time (Figure 2, AG News leave-one-out): WSD 1.45 min, Z-PEFT 30.7 min (extraction 26.8 min, selection and fitting 3.9 min), PEFTGuard 24.00 h, nearly three epochs. Inference: 0.67 min, 13.6 min plus 1.1 seconds, about 1.22 h. The authors state that full inference for Z-PEFT is about 5.4× faster than PEFTGuard; WSD is about 20.3× faster than Z-PEFT and 109.3× faster than PEFTGuard.
• Figure 3 gives WSD .445 when RIPPLE is held out, below the random ordering baseline of .5; Z-PEFT is .926. The authors state this is 48.08 AUROC points higher. A float32 sample of a rank-256 Llama-2 7B LoRA is about 4 GiB; the PEFTGuard representation for 850 training samples is about 3.3 TiB. The Z-PEFT feature file is 164.1 MiB, which the authors state is consistent with a reduction of about 21,000×.
• Table 5 in-domain: Flan-T5-XL and Qwen-1.5-7B are 1.0000/1.0000; Llama-2-13B is .9980/.9600; Llama-2-7B is .9986/.9794; RoBERTa-base is .9908/.9400. The authors state that the first three share Toxic Backdoors–Hard, rank 256, query/value; RoBERTa-base is IMDb, InsertSent, rank 16.
• Table 6 leave-one-out rank (Toxic Backdoors–Hard): AUROC for ranks 8 to 1024 is not below .9808, ACC above .9798; rank 2048 is .9600/.9200. The authors state it does not decrease monotonically with rank.
• Table 7 leave-one-out adapter: DoRA and LoRA+ are 1.0000/1.0000; LoRA .9956/.9720; QLoRA .9924/.9720; AdaLoRA .2628/.2880. The authors state that AdaLoRA is below random, with the ordering largely reversed, and attribute this to adaptive rank allocation.
• SHAP: attributions across layers, heads, and projections are relatively uniform; the appendix states that the value projection has the largest share globally and in several attack families. RIPPLE mainly uses energy-concentration and kurtosis, while stable rank is more important for AdaLoRA Hard. The authors state that no single joint group exceeds about .17 log-odds; the top 10%, 20%, and 50% of groups account for about 55%, 77%, and 95% of attribution. In Figure 7, AdaLoRA rank 8 true positive stable rank is +4.84 log-odds, output 6.75; LoRA rank 1024 outputs are −15.55 and 16.26; QLoRA rank 256 false negative output is −.04. The authors state this is not a causal relationship between spectral features and backdoors.

Limitations and future directions noted by the authors

• PADBench does not cover all variations of adapters in practice; naturally occurring backdoors, or attack mechanisms very different from the benchmark, may yield different performance (Limitations).
• PADBench only provides query and value attention projections; the evaluation does not establish whether spectral features generalize to other attention or MLP projections (Limitations).
• The reported accuracy uses the number of known backdoor test samples to select the threshold and should be viewed as score separability, not deployment accuracy; practical use requires calibrating the threshold under unknown attack prevalence (Limitations).
• Adaptive adversaries that optimize adapters to mimic benign spectral statistics are not evaluated; robustness against attacks that know the detector is a future direction (Limitations).

Experimental coverage

• The main experimental architecture is Llama-2 7B; Table 5 also includes in-domain results for Flan-T5-XL, Qwen1.5-7B, Llama-2-13B, and RoBERTa-base, with the authors noting that the experimental support for the latter two differs from the first three.
• Baselines are PEFTGuard and WSD. The data is PADBench, which the authors state contains 13,300 LoRA adapters; the heterogeneous setting uses LoRA, AdaLoRA, DoRA, LoRA+, and QLoRA, with ranks from 8 to 2048.
• Attack transfer covers InsertSent, RIPPLE, Syntactic, and StyleBkd; Table 3 also reports Alpaca and Hard-Rank256. The supplementary material includes leave-one-out rank (Table 6) and leave-one-out adapter method (Table 7).
• Metrics are AUROC and ACC with thresholds based on the number of known positive samples. Labels come from whether the adapter is backdoored. The paper does not report sample sizes per configuration, number of repetitions, or number of cross-validation folds.
• The appendix uses SHAP to analyze the trained detector. The paper states that adaptive adversaries mimicking benign spectral statistics are not evaluated, and that the weights only contain query and value projections.

Z-PEFT is a backdoor detector for read-only PEFT adapter weights, targeting attacks and configurations not seen during training.

When open repositories distribute adapters, closed-set detectors may not recognize new attacks. The method extracts singular value spectra by layer, projection, and attention head, and uses ℓ2 logistic regression to distinguish benign from backdoored, without running the base model or using triggers or fine-tuning data.

• In Table 2 (AG News leave-one-out attacks, default Llama-2 7B), Z-PEFT's AUROC/ACC are InsertSent .949/.931, RIPPLE .926/.893, Syntactic .910/.904, StyleBkd .989/.968, all higher than PEFTGuard and WSD in the same table. The authors state that average AUROC goes from .8395 for the in-budget PEFTGuard to .9433.
• In single-attack transfer, the authors state that Z-PEFT is higher than PEFTGuard on 8 of 12 pairs, with average AUROC going from .8153 to .8718. In Table 3, Alpaca is .635/.640; Table 4 in-domain aggregate is .998572/.979412.
• In Table 7, when AdaLoRA is held out, AUROC is .2628, which the authors state is below random. In Figure 2, Z-PEFT trains in 30.7 minutes, while PEFTGuard uses the full 24 hours; the authors state that full inference is about 5.4× faster.
• The authors state that they achieve state-of-the-art in the evaluated settings, and use per-head spectral features to balance structure and scale. They also interpret ACC based on known positive samples as separability, and note that adaptive adversaries mimicking benign spectra were not tested. The authors state that code will be released upon acceptance.