Fine-Pruning: Defending Against Backdoor Attacks on Deep Neural Networks

1. 01What problem does this paper attempt to solve?

   How to defend against DNNs with backdoors implanted during outsourced training: the authors propose and evaluate pruning, fine-tuning, and their combination, fine-pruning.

   Core problem: defending deep neural networks implanted with backdoors (backdoor / neural trojan) in outsourced training scenarios.

   • Scenario and importance: DNN training is costly and is often outsourced to third parties such as MLaaS; the authors state that an attacker can return a model that behaves normally on clean inputs but misclassifies in a targeted or random way when a trigger is present, and can fully control the trigger form.
   • Shortcomings of existing methods: the authors state that prior defenses against DNN backdoors are few, that countermeasures such as Chen et al.'s limited retraining were considered ineffective by their own authors, and that Liu et al.'s mitigation measures were tested only on MNIST; the authors state that this paper is the first defense that is fully effective on real models (statement from the original paper).
   • Core observation: based on Gu et al.'s observation—the backdoor exploits "backdoor neurons" that are dormant on clean inputs (spare capacity)—the authors reproduce this phenomenon on three attacks (Figure 4).
   • Paper's approach: evaluate two defenses, pruning and fine-tuning, and find that each is insufficient against sophisticated attackers; then design a pruning-aware attack and propose the fine-pruning defense combining pruning + fine-tuning.
   • Threat model (Section 2.2):
     • Attacker's goal: the returned model produces targeted or random misclassification on inputs containing the trigger while maintaining validation accuracy (otherwise the user would not deploy it).
     • Attacker's knowledge: strong white-box, full control over the training process and training data, but no access to the user's validation set.
     • Attacker's capabilities: can add any number of poisoned samples, modify clean samples, adjust training hyperparameters, and even manually set weights.
     • Victim system and defender: the user outsourcing training; the defender holds a clean validation set and can execute the model, prune neurons, or fine-tune with clean data.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three categories: classical ML poisoning attacks, DNN backdoor attacks, and previous limited defense attempts.

   Poisoning and evasion attacks on classical machine learning

   • Barreno et al. (2006)—proposed a framework for classifying ML attacks along three axes: integrity/availability, exploratory/causative, and targeted/indiscriminate.
   • Early work—exploratory attacks on intrusion detection systems and spam filters, and training-data poisoning attacks on spam filtering and intrusion detection; Suciu et al. (2018) proposed a unified modeling framework and StingRay, a targeted poisoning attack against various models (including CNNs).
   • Liu et al. (2017)—robust linear regression under noisy and poisoned data by recovering a low-rank subspace of the feature matrix (defense direction).

   DNN backdoor attacks (the three attacks reproduced in this paper)

   • Gu et al. (2017, BadNets)—backdoor attack on traffic sign recognition; this paper reproduces its untargeted attack; its empirical observation (backdoor activates dormant neurons) is the basis for this paper's pruning defense.
   • Chen et al. (2017)—targeted poisoning backdoor attack on face recognition (sunglasses trigger), reproduced in this paper.
   • Liu et al. (2018, NDSS)—targeted trojan attack on speech recognition, where the attacker cannot access training data and can only modify the trained model; this paper reproduces its attack but assumes a stronger attacker.
   • Muñoz-González et al. (2017) and Liu et al. (2017, Neural Trojans)—the former proposed a gradient-based method for generating poisoned data, and the latter studied neural trojans on an MNIST toy example and evaluated several mitigation techniques.

   Existing defense attempts (criticized by name in the paper)

   • Chen et al. (2017)—examined countermeasures including limited retraining with a held-out validation set, and the authors themselves concluded they were ineffective.
   • Liu et al. (2018)—pointed out that targeted backdoors disproportionately reduce accuracy on the target class and suggested using this as a detection method.
   • Liu et al. (2017)—the mitigation measures were tested only on MNIST, and the authors consider MNIST unrepresentative of real computer vision tasks.

   Baselines and positioning: this paper does not compare against defense baseline methods; the comparison is among four defender strategies—"no defense / pruning only / fine-tuning only / fine-pruning"—applied to the three reproduced attacks. Authors' positioning: to their knowledge, this paper is the first DNN backdoor defense that is fully effective on real models, and the first systematic analysis of attacker-defender interaction in backdoor attacks (both are statements from the original paper).

   Collapse this answer

3. 03How does the paper address this problem?

   First prune neurons that are dormant on clean inputs, then fine-tune on clean data to recover accuracy; the combination is fine-pruning.

   Overall approach: use the observation that "backdoor neurons are dormant on clean inputs" to perform pruning; for pruning-aware attacks that can bypass pruning, combine with fine-tuning to form the fine-pruning defense.

   Problem setting and formalization

   • A DNN is represented as a parameterized function F_Θ : ℝ^N → ℝ^M, with the i-th layer computed as a_i = ϕ_i(w_i a_i−1 + b_i); a ReLU output greater than zero is called "active", and equal to zero is called "dormant".
   • The defender receives the model returned by the attacker, has only a clean validation set, does not know the trigger form, and cannot change the model architecture (the attacker cannot change hyperparameters, only weights).

   Pruning defense

   • Procedure: run the network forward on the validation set and record the average activation of each neuron; prune neurons one by one in ascending order of average activation, recording accuracy at each step; stop when validation accuracy drops by more than a predetermined threshold.
   • Pruning location: prune only the last convolutional layer (conv3 of DeepID, conv5 of AlexNet and F-RCNN), on the grounds that later layers sparsely encode features from earlier layers, and pruning later layers has a greater effect on network behavior.
   • Three-stage observation: first prune to neurons that are not activated by either clean or backdoor inputs (no effect), then prune to neurons activated only by the backdoor (attack success rate drops, clean accuracy unchanged), and finally prune to neurons for clean inputs (accuracy drops, defense terminates).

   Pruning-aware attack (designed by the authors to test the defense)

   • Four steps (Figure 7): 1) train on clean data; 2) prune dormant neurons (the number is a design parameter); 3) retrain the pruned network on poisoned data, and if capacity is insufficient, restore one neuron and retrain until both clean accuracy and attack success rate are satisfactory; 4) "de-prune": restore all pruned neurons and their original weights, and lower their biases so that they remain dormant on clean inputs.
   • Effect: clean and backdoor behavior are compressed onto the same set of neurons; the restored neurons act as "decoys", and when the defender prunes, they are pruned first, so backdoor behavior is unaffected.

   Fine-tuning and fine-pruning

   • Fine-tuning: initialize with the weights returned by the attacker, use a smaller learning rate, and continue training on clean data; the authors report that AlexNet fine-tuning converges within one hour, whereas training from scratch can take more than six days.
   • Why fine-tuning fails (authors' explanation): in the baseline attack, backdoor neurons are dormant on clean inputs, so their gradients are zero and fine-tuning does not update their weights.
   • fine-pruning: prune first, then fine-tune. For the baseline attack, pruning removes the backdoor neurons and fine-tuning recovers the accuracy loss caused by pruning; for the pruning-aware attack, pruning removes only the decoy neurons, but because backdoor and clean behavior share neurons, subsequent fine-tuning updates the weights encoding the backdoor, thereby eliminating the backdoor.

   Success criteria and evaluation method

   • Metrics: clean-input classification accuracy (cl) and backdoor attack success rate (bd); the success rate of a targeted attack is defined as the proportion misclassified as the target class, and for an untargeted attack (traffic signs) it is defined as 1 − A_backdoor/A_clean.
   • Comparison setting: three defender strategies—no defense, fine-tuning only, and fine-pruning—× two attacker strategies—baseline attack and pruning-aware attack—evaluated on three tasks (Table 1); and a game analysis using "clean accuracy − attack success rate" as the defender's utility (Table 2).

   Collapse this answer

4. 04What experiments did the paper conduct?

   The attack is reproduced on three tasks and four defense strategies are evaluated; fine-pruning reduces the success rate of most attacks to 0–2%.

   Experimental setup

   • Networks tested: DeepID (face recognition), AlexNet (speech recognition), Faster-RCNN (traffic sign detection), all ReLU convolutional networks.
   • Datasets: YouTube Aligned Face (1283 people, 100 images each, 90% for training); speech dataset (3000 training samples, 300 per digit, 1684 test samples); U.S. traffic signs (6889 training, 1724 test images).
   • Attack reproduction: the face attack poisons images of 180 people, clean accuracy 97.8%, attack success rate 100%; the speech attack adds 300 poisoned samples, clean accuracy 99%, success rate 77%; the traffic sign attack adds one poisoned version to each training image, clean accuracy 85%, success rate 99.2%.
   • Defender strategies: no defense, pruning only, fine-tuning only, fine-pruning; pruning prunes only the last convolutional layer, with a stopping threshold of a 4% drop in clean accuracy; fine-tuning trains on clean data until convergence.
   • Metrics: clean-input classification accuracy (cl) and backdoor attack success rate (bd); traffic signs are an untargeted attack, with success rate defined as 1 − A_backdoor/A_clean.
   • Hardware: a cluster of NVIDIA P40, P100, K80, GTX 1080; the paper does not report the number of repetitions.

   Main results

   Table 1: clean accuracy (cl) and attack success rate (bd) under the three defense strategies, with one group each for the baseline attack and the pruning-aware attack.

   Swipe horizontally to read the table

   Task	Attack	No defense cl/bd	Fine-tuning cl/bd	fine-pruning cl/bd
   Face recognition	Baseline	0.978/1.000	0.978/0.000	0.978/0.000
   Face recognition	pruning-aware	0.974/0.998	0.978/0.000	0.977/0.000
   Speech recognition	Baseline	0.990/0.770	0.990/0.435	0.988/0.020
   Speech recognition	pruning-aware	0.988/0.780	0.988/0.520	0.986/0.000
   Traffic signs	Baseline	0.849/0.991	0.857/0.921	0.873/0.288
   Traffic signs	pruning-aware	0.820/0.899	0.872/0.419	0.874/0.366

   • The authors state that in the worst case fine-pruning reduces clean accuracy by only 0.2%, and in some cases slightly improves it.
   • The authors state that for targeted attacks (face, speech), fine-pruning reduces the attack success rate to 0% in most cases, with the worst being 2% for the speech baseline attack.
   • For the untargeted traffic sign attack, the authors note that 28.8% and 36.6% may appear high, but in an untargeted attack any misclassification counts as a successful attack, making defense harder.

   Separate evaluation of the pruning defense

   • What was tested: the effect of pruning only (last convolutional layer) on the three baseline attacks (Figure 6a/c/e).
   • Results: with a 4% drop in clean accuracy as the stopping condition, the attack success rates for face, speech, and traffic signs dropped from 99% to 0%, 77% to 13%, and 98% to 35%, respectively; the pruning proportion threshold at which the backdoor is disabled is between 0.68× and 0.82× the total number of neurons.
   • Authors' interpretation: pruning is computationally cheap and requires only forward execution, but the threshold is unknown to the defender, and it suggests the possibility of a pruning-aware attack.

   Evasion of pruning by the pruning-aware attack

   • What was tested: the performance of models generated by the pruning-aware attack under the pruning defense (Figure 6b/d/f), with activation distributions in Figure 8.
   • Results: on the face model, pruning first reduces clean accuracy, and by the time the attack success rate begins to drop, clean accuracy is already below 23%; on the speech model, reducing the attack success rate to 13% requires a 55% drop in clean accuracy (the baseline attack requires only 4%); on traffic signs, pruning reduces clean accuracy but the attack success rate remains high. The attack compresses clean and backdoor activations into 3%–15% of the neurons in the last convolutional layer (traffic, speech), and about 84% of neurons in the face model are dormant for both types of input.
   • Authors' interpretation: clean and backdoor behavior can be mapped onto the same set of neurons, and the pruning defense is ineffective against sophisticated attackers.

   Game analysis

   • What was tested: using "clean accuracy − attack success rate" as the defender's utility, comparing two attacks × two defenses, fine-tuning/fine-pruning (Table 2 gives the speech task: fine-tuning 0.555 against baseline, 0.468 against pruning-aware; fine-pruning 0.968 against baseline, 0.986 against pruning-aware).
   • Results and authors' interpretation: the defender's optimal strategy is always fine-pruning; the authors state that the utility matrices for the speech and traffic sign tasks lead to the same conclusion (values not given).

   Other ablations and analyses

   • Fine-tuning cost: the authors report that fine-tuning converges within minutes on high-end GPUs, AlexNet fine-tuning completes within one hour, whereas training from scratch can take more than six days.
   • Activation visualization: Figure 4 shows that after the baseline attack, the face network's last convolutional layer contains neurons activated only by backdoor inputs; Figure 8 shows that after the pruning-aware attack, both types of input activate the same set of neurons.
   • Exception: the traffic sign pruning-aware attack has a success rate of 89.9% with no defense, slightly lower than the baseline attack's 99.1% (Table 1); the paper does not explain why.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out two open problems: architecture coverage and stronger attacks; the experiments cover only three tasks and three CNNs.

   Limitations and future directions noted by the authors

   • Architecture applicability (Section 4.1 Threats to Validity): all three attacks are based on ReLU-activated convolutional networks; backdoor attacks on architectures such as RNNs and LSTMs have not been sufficiently studied, and the authors cannot be certain whether the defense applies to all deep networks.
   • No strong guarantee (Section 4 Discussion): the authors cannot guarantee that an attacker cannot find an initial parameter Θi near which a local minimum containing the backdoor still exists, thereby resisting fine-pruning.
   • Noise-adding variant (Section 4 Discussion): the authors propose a stronger but more computationally expensive variant—adding noise to the parameters before fine-tuning—and consider "how much noise is needed" a question worthy of future research.
   • Not the final word (Section 4 Discussion): the authors explicitly state that they cannot guarantee this defense is the final answer to DNN backdoor attack and defense.

   Scope of experiments

   • The networks tested are DeepID, AlexNet, and Faster-RCNN, 3 in total, all ReLU convolutional networks (Section 4.1).
   • The evaluation tasks are face recognition, speech recognition, and traffic sign detection, 3 in total; the attack types cover both targeted (face, speech) and untargeted (traffic sign).
   • The defense evaluation includes four strategies: no defense, pruning only, fine-tuning only, and fine-pruning; the attacker includes baseline and pruning-aware (Table 1).
   • Pruning acts only on the last convolutional layer of each network (Section 3.1).
   • The paper does not report the number of repetitions for each experiment, nor the amount of clean data used for fine-tuning or the specific hyperparameters.

   Collapse this answer

6. 06Summarize the main content of the paper

   For DNN backdoors in outsourced training, the authors propose the pruning + fine-tuning fine-pruning defense and evaluate it systematically.

   A paper on DNN backdoor defense (2018) studying the outsourced training scenario: a model returned by an untrusted third party behaves normally on clean inputs but misclassifies when a trigger is present.

   • Problem: the authors state that prior DNN backdoor defenses for real models are essentially absent, and existing countermeasures are either considered ineffective by their own authors or validated only on MNIST.
   • Method: reproduce three backdoor attacks—face (DeepID), speech (AlexNet), traffic signs (F-RCNN); based on the observation that "backdoor neurons are dormant on clean inputs", propose a pruning defense; then design a pruning-aware attack that makes clean and backdoor behavior share neurons to bypass pruning; finally propose fine-pruning, which prunes first and then fine-tunes on clean data.
   • Key results (Table 1): pruning or fine-tuning alone cannot counter both attackers simultaneously; fine-pruning reduces the face recognition attack success rate to 0% (baseline and pruning-aware), speech recognition to 2% (baseline) and 0% (pruning-aware), and the untargeted traffic sign attack to 28.8% (baseline) and 36.6% (pruning-aware), with clean accuracy dropping by at most 0.2%.
   • Game analysis: using "clean accuracy − attack success rate" as utility, on the speech task fine-pruning's utility against the two attackers (0.968, 0.986) is higher than fine-tuning's (0.555, 0.468); the authors state that the defender's optimal strategy is always fine-pruning (Table 2).
   • Authors' conclusion: fine-pruning can weaken or even eliminate the backdoor in all attacks they evaluated, and is a first step toward secure outsourced training; the authors also note that the defense is currently validated only on ReLU convolutional networks and cannot rule out stronger attacks that resist fine-pruning.

   Collapse this answer

Related papers