Stealthy backdoors as model compression artifacts

1. 01What problem does this paper attempt to solve?

   Whether a malicious trainer can hide a backdoor as a product of compression: the uncompressed model cannot be detected, and it is activated only after compression.

   Core question: Does model compression (pruning, quantization) give attackers an opportunity to inject a stealthy backdoor—the uncompressed model passes detection, and the backdoor only takes effect after compression and deployment?

   • Scenario and motivation: Large models are often compressed before being deployed to resource-constrained devices. The supply-chain scenario the authors envision is: a malicious trainer releases a specially crafted model, a model repository uses state-of-the-art methods to detect backdoors and then accepts it, and the deployer downloads, compresses, and deploys it, trusting the repository's detection and performing no further security testing (Figure 1).
   • Shortcomings of existing methods: The authors argue that current backdoor detection is all performed on uncompressed models, whereas compression changes model behavior; deployers typically only test post-compression accuracy and do not perform dedicated backdoor detection, leaving a gap between detection and use.
   • Core observation: Model behavior differs before and after compression, and a malicious trainer may hide undesirable behavior in differences that only appear in the compressed form; the authors call this the first work to use compression to hide backdoors.
   • What the paper proposes: Stealthy backdoor attacks targeting two common compression techniques, 8-bit quantization and auto-compress pruning, with stealthiness evaluated using MNTD and Neural Cleanse; it also gives a simple defense—detect on the actual post-compression deployment form.
   • Threat model (Section III):
     • Attacker goal: The trained model has no backdoor and is undetectable when uncompressed, but after compression, triggered inputs are misclassified into the target class.
     • Attacker knowledge: Knows the deployer will use common compression methods; for quantization, does not know the specific samples in the calibration set; for pruning, knows the auto-compress tool and the base pruning method, and has two levels of knowledge about the overall pruning rate: "known" and "only knows the range".
     • Attacker capability: Full control over the training process; does not control the compression process or calibration data.
     • Victim systems: The testers of the model repository (who use state-of-the-art detection and do not know the model will be compressed) and the model deployers who download and deploy the compressed model.

   Collapse this answer

2. 02What related research exists?

   Related work covers three categories: model compression, backdoor attack/defense and evasion, and compression security risks; the authors state that this paper is the first work to use compression to hide a backdoor.

   Model compression

   • Quantization: 8-bit quantization and QAT of Migacz et al. [6] and Jacob et al. [7]; Krishnamoorthi et al. [18] report 2–3× speedup of INT8 on mobile CPUs. The attacks in this paper target 8-bit quantization.
   • Pruning: Unstructured [8]–[12] and structured pruning [13]–[17], as well as auto-compress [17], [44] that automatically determines per-layer pruning rates using reinforcement learning or heuristic search; Liu et al. [17] report an 8× speedup on Samsung Galaxy S10. This paper assumes the deployer uses a popular auto-compress tool.

   Backdoor attacks and defenses

   • Injection methods: Gu et al. [24] inject backdoors by poisoning the training set; Liu et al. [23] inject into a pretrained model when no training data is available. This paper assumes the malicious trainer has full control over training and adopts a poisoning approach similar to Gu et al., but changes the loss function to create a backdoor in the compressed artifact.
   • Detection methods: Trigger pattern reconstruction class—Neural Cleanse [32], DeepInspect [35], Tabor [46], ABS [34]; NeuronInspect [47] distinguishes using saliency maps; meta-model class—MNTD of Xu et al. [48], which the authors call the current state-of-the-art. This paper uses Neural Cleanse (because it is limited to small trigger patterns) and MNTD for detection evaluation.
   • Evading detection: The latent backdoor of Yao et al. [49] only takes effect after transfer learning; Tang et al. [50] insert an additional Trojan module; Salem et al. [51] vary the trigger pattern and location. The authors argue that these works expose the limitations of existing defenses against adaptive attackers, whereas this paper (perhaps optimistically) assumes strong defenses exist and the attacker needs to hide further.

   Security concerns related to compression

   • Xiao et al. [52] use image scaling to generate poisoned samples that look normal; their threat model has the defender inspect the training set, which differs from this paper.
   • Ye et al. [53] find that pruning reduces model robustness to adversarial examples and propose a pruning method that preserves robustness.

   Baselines and benchmarks: The detection baselines are MNTD and Neural Cleanse (both reproduced according to the original papers' settings); the attack comparison is a conventional poisoning backdoor that does not consider compression (trained according to Chen et al. [45], Gu et al. [24]). The datasets are CIFAR-10 and GTSRB.

   Authors' positioning: It has previously been pointed out that compressed artifacts can be exploited by attackers, but the authors claim this paper is the first work to use compression to hide backdoors, and emphasize that it assumes stronger defenses and that the attacker does not know the specific compression details.

   Collapse this answer

3. 03How does the paper address this problem?

   Jointly train the uncompressed model (no backdoor) and the dynamically generated compressed model (with backdoor), then use clean-model distillation to reduce detectability.

   Overall idea: During training, the malicious trainer simultaneously optimizes the uncompressed model f and the compressed model f̂ dynamically generated at each training step, so that f predicts both clean and triggered inputs correctly and f̂ misclassifies triggered inputs into the target class; soft-label distillation from the clean model then brings f's decision boundary close to that of a normal model, reducing the probability of detection. The method is named compression artifact backdoor and has two variants, standard and distilled.

   Attack goal and overall loss

   • Goal formalization: For all clean inputs x, f(x)=y and f̂(x)=y; for triggered inputs xtr, f(xtr)=y while f̂(xtr)=t (t is the target class).
   • The total loss is a weighted sum of the uncompressed model loss and the compressed model loss: loss(D̂) = ∑ [loss_f(x,y) + α · loss_f̂,t(x,y)], where α trades off the two terms.
   • The uncompressed model loss is loss_f(x,y) = (1−β) · l(f(x), y) + β · l(f(xtr), y), encouraging the uncompressed model to classify inputs correctly with and without the trigger; the compressed model loss classifies clean inputs correctly and triggered inputs into t, with weight γ. The trigger pattern is a simple fixed white square.

   Attack against quantization

   • To remain robust to an unknown calibration set, only the difference in model weights before and after quantization is used, ignoring the difference in activations: during training only the weights are quantized to generate the compressed model, even though the deployer quantizes both weights and activations.
   • QAT techniques are borrowed: FP32 computation is used to simulate 8-bit integer computation, and the Straight-Through Estimator is used to solve the zero-gradient problem of the round operation.

   Attack against pruning

   • Known pruning rate: The attacker first trains a clean model and uses auto-compress with the known overall pruning rate to obtain per-layer pruning rates for training; if the per-layer pruning rates obtained by auto-compress on the trained model are inconsistent with the input, the per-layer pruning rates output in the previous round are iteratively used as the input for the next round until they match. The authors report that most models were already close in the first round, and a few required several iterations.
   • Unknown pruning rate: At each training step, n compressed models with different pruning rates are generated, and the loss is averaged (Equation (5) in the paper). The experiments use n=3: the lower bound of the pruning range, a random value within the range (resampled each step), and the upper bound; the per-layer pruning range is obtained by running auto-compress on the clean model at the lower and upper bounds of the overall pruning rate.

   Distilled attack

   • Motivation: The standard attack does not consider detection, and the model may have detectable differences from a normal model; the authors' intuition is that the closer the uncompressed model's decision boundary is to the clean model, the harder it is to detect.
   • Approach: Use the prediction vector (soft labels) of a pretrained clean model fc instead of one-hot labels to compute the loss; and at each step use gradient ascent to modify training samples so as to maximize the prediction difference between f and fc, adding these samples to training as well (Equation (7)).

   Implementation and evaluation method

   • Implementation: PyTorch; quantization uses PyTorch QAT (VGG-16 has bias disabled because QAT does not support Conv2D bias, and Appendix B states this does not affect accuracy), with the backend preferring ARM (QNNPACK); pruning uses l1-norm-based filter-level structured pruning [15], and the auto-compress tool is the simulated-annealing-based version in NNI.
   • Success criterion: An attack succeeds if and only if the uncompressed model passes detection and the backdoor is effective after compression; the metrics are Accuracy, Triggered Accuracy (the proportion of triggered images still predicted correctly, where a high value indicates the backdoor is not activated), and Attack Success Rate (the proportion of triggered images classified into the target class).

   Collapse this answer

4. 04What experiments did the paper conduct?

   For the quantization and pruning attacks, post-compression ASR exceeds 90% in most settings, the uncompressed model has little accuracy loss, and MNTD and Neural Cleanse mostly fail to detect them.

   Experimental Setup

   • Models under test: VGG-16, ResNet-18, MobileNet (version 2), implemented in PyTorch.
   • Datasets: CIFAR-10 (50,000 training / 10,000 test, 10 classes); GTSRB (39,208 training / 12,630 test, 43 classes). Three settings for the quantization calibration set: 1,000 random images from the training set; similar distribution (a CIFAR-100 subset for CIFAR-10, a TSRD subset for GTSRB); different distribution (an SVHN subset), 1,000 images each.
   • Trigger and target classes: white square trigger; for each architecture × dataset, 5 backdoored models with different target classes were trained and the mean and standard deviation reported; additionally 10 clean models were trained as a baseline.
   • Hyperparameters: quantization attack α=1.0, β=0.5, γ=0.5; pruning attack α=1, β=0.5, γ=0.9; the known pruning rate experiment reports 0.3, and the unknown pruning rate target range is [0.3, 0.5].
   • Detection evaluation: MNTD follows the original paper's setting, training 4,096 shadow models (2,048 clean) and averaging 5 meta-classifiers; the metrics are the AUC for distinguishing clean from backdoored models and the TPR at FPR=0.1; Neural Cleanse trigger reconstruction is repeated 3 times and the minimum taken, reporting the AUC of the maximum anomaly index (the target-class AUC in parentheses).
   • Control: a conventional poisoning backdoor (without considering compression) as the baseline, with 10 trained per setting.

   Main Results

   Attack Success of the compressed models for the quantization distillation attack (Table I, calibration set from the training set) and the pruning distillation attack with known pruning rate 0.3 (Table III):

   Swipe horizontally to read the table

   Dataset	Model	Quantization ASR (Table I)	Pruning ASR (Table III)
   CIFAR-10	VGG-16	99.7 ± 0.2	89.5 ± 8.3
   CIFAR-10	ResNet-18	80.0 ± 37.2	96.8 ± 1.5
   CIFAR-10	MobileNet	99.8 ± 0.1	91.0 ± 8.9
   GTSRB	VGG-16	88.1 ± 19.8	92.8 ± 6.9
   GTSRB	ResNet-18	99.9 ± 0.1	99.4 ± 0.3
   GTSRB	MobileNet	73.9 ± 31.9	99.2 ± 0.2

   • The authors state that the clean accuracy of the uncompressed backdoored models drops by at most 0.7% relative to the clean models, and that accuracy on triggered images is close to clean accuracy, i.e., the backdoor is not activated in the uncompressed form; after compression, clean accuracy drops by at most 0.5% for quantization and at most 2.3% for pruning (VGG-16/CIFAR-10 drops 4%, which the authors state is still within the typical range for compression).
   • The authors state that the quantization attack success rate exceeds 99% in over half the settings and is above 73% in all settings; the pruning attack exceeds 89% on CIFAR-10 and 92% on GTSRB.
   • Settings with large standard deviation (e.g., ResNet-18 quantization 80.0 ± 37.2) indicate large variation across target classes; the paper does not explain the reason.

   Effect of Calibration Set and Pruning Rate Uncertainty

   • Unknown calibration set (Table II): under similar distribution the ASR is close to the same-source setting; under different distribution (SVHN) the ASR drops, with CIFAR-10 at 94.7 ± 9.3 (VGG-16), 68.7 ± 34.0 (ResNet-18), 96.1 ± 4.1 (MobileNet), and GTSRB at 56.5 ± 37.3, 98.9 ± 1.2, 67.8 ± 32.6; the authors state it remains effective most of the time (CIFAR-10 >68%, GTSRB >56%).
   • Unknown pruning rate (Figure 3): the authors state that within the target range 0.3–0.5 the ASR is above 89% in all cases, with the sole exception of VGG-16 on GTSRB at 65% when the target range is 0.3; outside the target range the success rate remains relatively high. Models trained with a known pruning rate (0.3) degrade quickly at other pruning rates.
   • Inconsistent base pruning method (Figures A2, A3): when the trainer uses l1-norm and the deployer uses l2-norm, the authors state the attack remains effective.

   Stealthiness Against Backdoor Detection

   • MNTD (Tables IV, V): the baseline conventional backdoor AUC exceeds 0.95 on ResNet-18/MobileNet (CIFAR-10) and exceeds 0.8 on all of GTSRB (the VGG-16 baseline is only 0.6, which the authors state is because the meta-classifier does not train properly in that setting). Except for VGG-16/CIFAR-10, the AUC of this paper's attacks is all lower than the baseline; on GTSRB the AUC of the standard attack is below 0.5 and often close to 0.0. At FPR=0.1, 8 of 36 configurations have a detection rate below 10%; for the distillation attack, all are below 50% except CIFAR-10 MobileNet under unknown pruning rate and quantization.
   • Neural Cleanse (Table VI): the baseline AUC exceeds 0.8 in all cases (ResNet-18 is 1.0 on both datasets); for this paper's attacks, 35 of 36 settings have an AUC below 0.8, and 9 settings have a detection rate of 0% at FPR=0.1 (Table A7); the target-class AUC in parentheses is lower, and the authors state that detection cannot identify the true target class.
   • Explanation for AUC below 0.5: the authors hypothesize that the meta-classifier is confused by the effect of "training for compression"; Table A6 shows that when the same meta-classifier is used to distinguish ordinary clean models from clean models trained for compression, the AUC on GTSRB is likewise far below 0.5, which the authors state supports this hypothesis.
   • Distillation vs standard: on CIFAR-10 pruning, distillation is stealthier (average detection rate at FPR=0.1 drops from 66% to 38%); on GTSRB the standard attack instead has a lower AUC; on quantization the two are similar, with the detection rate above 50% in over half the settings, and the authors state that reducing the detection rate of the quantization attack is left for future work.

   Post-Compression Detection (Defense Validation) and Other Analyses

   • Post-compression detection (Table VII, CIFAR-10 distillation attack models): MNTD has an AUC of 1.00 in 5 of 9 settings, and at least 0.89 in the rest except VGG unknown pruning rate (0.68 ± 0.21); Neural Cleanse has an AUC greater than 0.78 in 5 of 6 settings. The authors state that detection on the actual deployment form yields reliable detection.
   • Other ablations: quantizing all layers vs only the later layers during training (Table A2), the former lowers ASR by 19–56% in 3 of 4 settings; disabling Conv2D bias in VGG-16 has little effect on accuracy (Table A1); the ASR of the standard attack is slightly higher than that of the distillation attack (Tables A3–A5).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors leave reducing the detection rate of the quantization attack and the impact of different pruning tools for future work; the experiments are limited to two vision datasets and three models.

   Limitations and Future Directions Noted by the Authors

   • The detection rate for quantized attacks remains high (more than half of the settings are above 50% at FPR=0.1); the authors state that reducing the detection rate under model quantization is left for future work (Section VIII-A).
   • The impact of trainers and deployers using different auto-compress tools and base pruning methods was only preliminarily experimented with; the authors leave a comprehensive analysis for future work (Section III).
   • In settings with a known pruning rate, the iterative process of matching layer pruning rates is not guaranteed to converge; the authors report that in experiments most models were already close in the first round, with a few requiring several iterations (Section IV-C).
   • MNTD's baseline detection fails on VGG-16/CIFAR-10 (AUC 0.6); the authors tried several hyperparameter sets but none made the meta-classifier work properly, and because meta-classifier training is very slow they could not test more (Section VIII-A footnote).
   • Detecting quantized models with Neural Cleanse faces a zero-gradient obstacle; the authors state that it might be solvable with QAT's STE but was not tested (Table VII note).

   Scope of Experiments

   • The models tested are VGG-16, ResNet-18, and MobileNet (version 2), 3 in total, with datasets CIFAR-10 and GTSRB (Section V).
   • Compression methods cover 8-bit quantization (PyTorch QAT, ARM/QNNPACK backends) and l1-norm-based filter-level structured pruning (NNI simulated annealing auto-compress); other quantization bit widths such as 16-bit are outside the experimental scope (Sections II-A, V).
   • Detection evaluation covers two methods, MNTD and Neural Cleanse; other defenses such as NeuronInspect and ABS are only discussed in related work (Sections II-C, VIII).
   • For each architecture × dataset, 5 backdoored models of different target classes and 10 clean models are trained; MNTD uses 4,096 shadow models and 5 meta-classifiers (Sections V, VIII-A).
   • Only a fixed white square is used as the trigger pattern; the paper mentions that optimized trigger patterns may be stronger but does not adopt them (Section IV-A).

   Collapse this answer

6. 06Summarize the main content of the paper

   Train the backdoor as a product of compression that is activated only after pruning or quantization; the uncompressed model can evade detection by MNTD and Neural Cleanse.

   An attack paper on model supply-chain security: a model released by a malicious trainer has no backdoor in its uncompressed form and can pass state-of-the-art detection, but the backdoor is activated after the deployer compresses it (pruning or quantization).

   • Problem: The model repository detects the uncompressed model, whereas what is actually deployed is the compressed model; the gap between detection and use can be exploited.
   • Method: During training, jointly optimize the uncompressed model (predicting both clean and triggered inputs correctly) and the compressed model dynamically generated at each step (assigning triggered inputs to the target class); the quantization attack uses QAT to simulate and quantizes only weights, while the pruning attack uses auto-compress to predict per-layer pruning rates and supports scenarios where only the pruning rate range is known; soft-label distillation from the clean model is then used to narrow the difference from a normal model.
   • Results: On three models on CIFAR-10/GTSRB, the quantization distilled attack ASR is 73.9%–99.9% (Table I), and 89.5%–99.4% when the pruning rate is known to be 0.3 (Table III); the uncompressed model's clean accuracy drops by at most 0.7%, and the backdoor does not activate on triggered inputs.
   • Robustness: When the calibration set distribution differs (SVHN), ASR decreases but remains above 68% on CIFAR-10 and above 56% on GTSRB (Table II); when the pruning rate is unknown, within the target range 0.3–0.5, ASR exceeds 89% in all cases except one at 65% (Figure 3).
   • Stealthiness: The detection AUC of MNTD and Neural Cleanse on the uncompressed backdoored model is almost entirely lower than the conventional backdoor baseline, and Neural Cleanse has a detection rate of 0% at FPR=0.1 in 9/36 settings; when detection is performed after compression, MNTD reaches an AUC of 1.00 in most settings (Table VII).
   • Authors' conclusion: Such attacks resemble time-of-check/time-of-use vulnerabilities, and security-critical applications must detect models in their actual deployment form; the authors state that reducing the detection rate of the quantization attack and analyzing the impact of different pruning tool combinations are left for future work.

   Collapse this answer

Related papers