Regula Sub-rosa: Latent Backdoor Attacks on Deep Neural Networks

1. 01What problem does this paper attempt to solve?

   The authors propose the latent backdoor: a backdoor embedded in the Teacher model is automatically transferred to all Student models through transfer learning.

   Core problem: Traditional DNN backdoor attacks require the attacker to control the Student model's training data or training process within a short window before deployment, which is difficult to achieve in practice; the authors investigate whether a backdoor can be embedded in a shared Teacher model and survive transfer learning.

   • Problem scenario: Most organizations today lack the resources to train large models from scratch, and instead download public Teacher models such as VGG16 and perform transfer learning with local data to obtain a Student model. The authors consider that the model provider step in this pipeline may be subject to attack.
   • Shortcomings of existing methods: According to the paper, traditional backdoor attacks (such as BadNets) require control over the Student's training data or training process, which is difficult to carry out unless the attacker is an employee responsible for training or a data provider; moreover, the backdoor can only target labels that already exist in the Teacher.
   • Core observation: Transfer learning freezes the first K layers of the Teacher; if the trigger is associated with the intermediate feature representation of these frozen layers rather than the final label, the backdoor can survive transfer learning; the Teacher does not contain the target label yt, which renders detection based on label scanning ineffective.
   • Method proposed in the paper: The latent backdoor attack—the attacker injects a "latent" trigger into the Teacher model for a label yt that does not yet exist and erases traces of yt, and the victim activates the backdoor themselves when adding the yt label during transfer learning. The authors validate on four tasks—digit recognition, traffic signs, faces, and iris—and three real-world scenarios, and evaluate 4 defenses.
   • Threat model:
     • Goal: Perform a targeted backdoor attack on a specific class yt, so that inputs with the trigger are misclassified by the Student model as yt.
     • Knowledge: The attacker knows the set of layers recommended for freezing at publication time (assuming the victim follows the recommendation); no access to any Student training data or training process is required.
     • Capability: Has sufficient compute to train or retrain the Teacher model, and can collect samples of yt; when the Teacher and Student tasks differ, additional non-target samples similar to the Student task need to be collected.
     • Victim system: A Student model obtained via transfer learning from the infected Teacher model and whose output labels include yt.

   Collapse this answer

2. 02What related research exists?

   Related work is divided into four groups: backdoor attacks, backdoor defenses, transfer learning, and data poisoning and other adversarial attacks.

   Backdoor attacks

   • BadNets (Gu et al., 2017/2019): Injects a backdoor by poisoning the training dataset, adding a trigger to some training images and changing their labels, so that the model performs normally on both clean and adversarial inputs.
   • The Trojaning attack of Liu et al. (2018): Constructs a trigger that elicits a large response on a specific neuron, establishing a strong association between the trigger and the neuron and reducing the amount of training data needed to inject the backdoor.
   • Other backdoor attacks: Chen et al. (2017) propose a backdoor for the restricted scenario where the attacker can only poison part of the training set; other work directly tampers with the hardware running the model (Clements & Lao 2018; Li et al. 2018). The paper points out that its attack differs from these in that it does not require access to the Student model, its data, or the hardware it runs on.

   Backdoor defenses

   • Neural Cleanse (Wang et al., 2019): Scans the model's output labels and reverse-engineers potential triggers, based on the fact that the perturbation required for a backdoor target label is much smaller than for clean labels; it requires some clean training data.
   • Activation Clustering (Chen et al., 2018): Detects injected backdoor data based on the different activation patterns of poisoned and benign inputs; it requires the complete training data (clean + poisoned).
   • Fine-Pruning (Liu et al., 2018): First prunes redundant neurons least useful for classification, then fine-tunes with clean data to restore performance. There are also three defenses by Liu et al. (2017) and spectral signature detection by Tran et al. (2018); the paper points out that these defenses all target ordinary backdoors and require poisoned training data, and are not applicable to the latent backdoor.

   Transfer learning and poisoning attacks

   • Transfer learning: Has been shown effective on vision, speech, and text tasks; Yosinski et al. (2014) compared the effect of different transfer approaches on performance, and Razavian et al. (2014) studied the correlation between Teacher–Student task similarity and performance.
   • Poisoning attacks: Poison training data to change the model's behavior on clean samples, without relying on a trigger; defenses mostly focus on cleaning the training set. The paper states that this approach is ineffective against backdoor attacks and impractical under this paper's attack model, because the defender cannot obtain the poisoned training set used for the Teacher.

   Baselines and benchmarks: The baseline for attack comparison is randomly generated triggers (Figure 5); the baselines for defense comparison are Neural Cleanse and Fine-Pruning (§7.1). The experiments use six datasets: MNIST, GTSRB, LISA, VGG-Face, PubFig, and CASIA IRIS (Table 1).

   The authors position this paper as a stronger backdoor variant: it targets the Teacher model, is undetectable on the Teacher, can propagate to multiple Student models through transfer learning, and can target future labels that do not yet exist.

   Collapse this answer

3. 03How does the paper address this problem?

   Associate the trigger with the intermediate feature representation of the Teacher's frozen layers, erase traces of the target label after injection, and rely on transfer learning for automatic activation.

   Overall idea: The attacker associates a trigger for the target class yt with the feature representation of an intermediate layer of the Teacher model (located within the first K layers to be frozen) rather than the final classification layer, then erases traces of yt from the model before publishing; when the victim adds the yt label during transfer learning, the backdoor is "activated by itself." The method is named the latent backdoor attack.

   Design insights

   • Associate the trigger with features rather than labels: Associate the trigger with the feature representation of clean yt samples at the Teacher's intermediate layer, decoupling trigger injection from the classification result, so that removing yt from the output labels afterward does not affect the trigger.
   • Inject into frozen layers: The attacker recommends freezing the first K layers in the publication documentation as per convention, and injects the trigger into layer Kt (Kt ≤ K), ensuring that transfer learning does not destroy the trigger.

   Teacher-side injection pipeline (5 steps)

   1. Adjust the Teacher to include yt: Retrain the Teacher with target data Xyt (clean samples of yt) and non-target data X \yt (general samples similar to the target task), and replace the last classification layer.
   2. Generate the trigger ∆: Given the trigger's position and shape (a square in the bottom-right corner of the image), optimize the trigger pattern and color so that the features of trigger-bearing inputs at layer Kt are close to those of clean yt samples. A poisoned sample is defined as A(x, m, Δ) = (1−m) ∘ x + m ∘ Δ, where m is a binary mask marking the trigger region.
   3. Inject the trigger: Optimize the Teacher weights, with loss J_θ(θ; x, y) = ℓ(y, F_θ(x)) + λ · D(F_{θ Kt}(A(x,m,Δ_opt)), ϕ_θ), i.e., the standard training loss plus the difference at layer Kt between poisoned samples and the yt feature representation (denoted ϕθ), with λ balancing the two terms; the difference is measured by MSE.
   4. Erase traces of yt: Swap the last classification layer back to the original Teacher's classification layer, so that the model's labels match the publication claims and evade label-scanning detection.
   5. Publish: Recommend freezing the first K layers (K ≥ Kt) in the publication documentation.

   Student-side activation

   • The victim downloads the infected Teacher and performs transfer learning as instructed, with the Student task including the yt label; the weights of the first Kt layers are copied and frozen, and the trigger propagates as a live backdoor (Figure 4).
   • The attack works the same as a traditional backdoor: attach the recorded trigger ∆ to any input, and the Student misclassifies it as yt.
   • The paper gives Lemma 1: If transfer learning freezes at least the first Kt layers and yt is one of the Student's labels, then with high probability the latent backdoor becomes a live backdoor in the Student (the proof is an argument based on the transfer of feature similarity).

   Key parameter choices

   • Choice of Kt: Kt cannot exceed the actual number of frozen layers K; the authors state that the practical strategy is to find the smallest Kt at which the injection optimization converges, then recommend freezing the first k ≥ Kt layers. In the experiments, the smallest effective Kt for the four tasks is the first fully connected layer (3 for Digit, 14 for Face/Iris) or the last convolutional layer (6 for TrafficSign).
   • Trigger configuration: A square mask located in the bottom-right corner of the input image, sized at 4% of the whole image; the authors state that a square is chosen to ensure the trigger is distinctive and does not occur naturally.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Multi-image attack success rates on four tasks are 96.6%–100%, single-image 46.6%–92.4%; effective in three real-world scenarios; only 1 of 4 defenses is effective.

   Experimental setup

   • Models tested: Digit uses a 4-layer CNN (2 Conv + 2 FC); TrafficSign uses an 8-layer CNN (6 Conv + 2 FC); Face and Iris use a 16-layer VGG-Face (13 Conv + 3 FC). All have a Teacher/Student structure (Table 1, Appendix Tables 6–8).
   • Datasets: Digit uses MNIST (Teacher recognizes 0–4, Student recognizes 5–9); TrafficSign uses GTSRB (Teacher, 39K training images) and LISA (Student, 17 classes, 3.65K); Face uses VGG-Face (Teacher, downsampled to 31 classes, 3K) and PubFig (Student, 65 classes, 6K); Iris uses CASIA IRIS (16K images, 1K people, Student 520 classes, 480 non-target classes). The four datasets X \yt, Xyt, X s, and X eval are mutually disjoint (Table 1).
   • Target data scale: Multi-image attacks |Xyt| ≤ 50 (Digit 45, TrafficSign 50, Face 45, Iris 3); single-image attacks |Xyt| = 1, run 20 times per task and averaged (16 times for Iris).
   • Metrics: Attack success rate (the proportion of trigger-bearing inputs on X eval classified as yt) and the Student model's classification accuracy on clean test data; the reference group is the accuracy of a Student transferred from a clean Teacher.
   • Trigger: A square in the bottom-right corner, occupying 4% of the image; the §5 experiments assume Kt = K.
   • Real-world scenarios: Traffic signs (10 photos of US Stop signs taken with a phone as Xyt), iris (5 cropped iris images from target headshots found via Google search, 160 iris photos taken as selfies by 8 lab volunteers as X s), politician faces (10 low-resolution Google headshots each for 9 targets; Student uses frames from YouTube congressional videos, 13 classes, 1.7K images).

   Main results

   Multi-image attacks (Table 2):

   Swipe horizontally to read the table

   Task	Attack success rate	Student accuracy with infected Teacher	Student accuracy with clean Teacher
   Digit	96.6%	97.3%	96.0%
   TrafficSign	100.0%	85.6%	84.7%
   Face	100.0%	91.8%	97.4%
   Iris	100.0%	90.8%	90.4%

   • The authors state that the attack is highly effective on all four tasks, and that the attacker infects the Teacher using no more than 50 target samples.
   • The authors point out that the Student accuracy obtained from the infected and clean Teachers is comparable, i.e., the attack does not harm model performance on clean inputs; among these, the Face task's accuracy after infection is 91.8%, lower than the clean Teacher's 97.4%.

   Single-image attacks (Table 4, average of 20 runs, 16 for Iris): Digit average success rate 46.6%, TrafficSign 70.1%, Face 92.4%, Iris 78.6%; the average Student accuracies with a clean Teacher are 96.0%, 84.7%, 97.4%, and 90.4%, respectively. The authors consider the decline in success rate to be expected (a single image makes it difficult to extract features accurately), and that the decline is more pronounced for the small model (Digit) than for the large models, which the authors attribute to the higher capacity of large models; Teachers actually used for transfer learning are mostly large models.

   Real-world scenario attacks (§6, Table 5)

   • Traffic signs: When using 16 photos of Stop signs taken with a phone as Xyt, the success rate is 100%; single-image attacks average 67.1% (model accuracy 87.4%).
   • Iris recognition: With 5 target images injected, the success rate is 90.8% (model accuracy 96.2%); single-image 77.1% (accuracy 97.7%).
   • Politician faces: Single-target multi-image attack success rate 99.8%, single-image 90.0%; for multi-target injection, the authors state that the success rate is close to 100% for up to 3 targets and drops to 60% at 9 targets, and Student accuracy is insensitive to the number of targets (Figure 8).

   Defense evaluation (§7, all on the Face task)

   • Neural Cleanse: Cannot be used on the Teacher (requires the yt label); when used on the infected Student, the reverse-engineered trigger differs noticeably from the actual trigger. The authors attribute its failure to the fact that the reverse engineering is based on end-to-end optimization from input to final label, and cannot discover manipulations that terminate at intermediate feature layers.
   • Fine-Pruning: Has no appreciable effect on the Teacher; when used on the Student, the success rate only begins to decline after pruning 25% of neurons, and the final model accuracy drops below 11.5%, which the authors consider impractical (Figure 9).
   • Input blurring (Gaussian filtering): The success rate declines faster than accuracy, but reducing the success rate below 20% requires reducing accuracy below 65%, which the authors consider too costly (Figure 10).
   • Fine-tuning more layers during transfer learning: With Kt=14 injection, as long as transfer learning fine-tunes any layer earlier than layer 14, the attack success rate drops to 0%; however, the Student does not know Kt, fine-tuning all layers conflicts with the goal of saving data in transfer learning, and with a fixed training set, fine-tuning more layers reduces accuracy (Figure 11).

   Other ablations and analyses

   • Necessity of trigger optimization (Figure 5, TrafficSign): 100 random triggers have a success rate < 20% and unstable accuracy; optimizing the trigger is effective.
   • Amount of non-target data: On Face, 8 of 31 classes suffice to reach a 100% success rate, and 2 classes give 83.6%; 4 images per class reach 100%, and 2 images give 93.1%.
   • Variation of (Kt, K) (Table 3, Face/Iris): The success rate remains 100.0%, and model accuracy varies slightly with (Kt, K) (e.g., Face 91.8% at Kt=14/K=14, 94.0% at Kt=15/K=15).
   • Failure of injection into earlier layers: The authors report that under the current implementation, lowering Kt to earlier convolutional layers causes the success rate to drop substantially, because the dimensionality of early convolutional layers is too large (about 25K for VGG-Face), causing the injection optimization to fail to converge.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that injection at earlier layers does not converge and is left to future work; experiments are limited to 4 image tasks and defenses were tested only on Face.

   Limitations and future directions noted by the authors

   • Injection into earlier layers unresolved: The authors report that under the current attack implementation, injecting the trigger into earlier convolutional layers often causes optimization to fail to converge due to the excessively large model dimensionality; the authors believe that attackers with more resources may overcome this with larger target/non-target datasets and compute, leaving it as future work (§5.2 Microbenchmark 3).
   • Declining success rate with multiple target injection: Injecting more targets requires injecting more triggers, making the injection optimization difficult to converge, and the success rate declines with the number of targets (§6.4).
   • Defense cost against stronger attackers: The authors point out that against a powerful attacker able to inject triggers into earlier layers, the multi-layer fine-tuning defense requires fine-tuning more or even all layers, at higher cost (§7.3).
   • Call for detection tools: The authors hope the work draws attention to the need for robust testing tools for DNNs to detect unexpected behaviors such as backdoors (§9 Conclusion).

   Scope of experiments

   • The models tested are 3 architectures (4-layer CNN, 8-layer CNN, 16-layer VGG-Face), covering 4 image classification tasks (Digit, TrafficSign, Face, Iris), all vision tasks (§5.1).
   • Single-image attacks are repeated 20 times per task (16 times for Iris) and averaged; the number of repetitions for multi-image attacks is not reported in the paper (§5.2, §5.3).
   • The §5 experiments assume Kt = K, and that the target data Xyt comes from the same source as the Student training data; different-source scenarios are tested in the three real-world cases in §6 (traffic signs, iris, politician faces).
   • Defense evaluation is conducted only on the Face task, including 4 defenses: Neural Cleanse, Fine-Pruning, input blurring, and multi-layer fine-tuning (§7).
   • The optimization hyperparameters used in the attack (such as the value of λ, number of training epochs, learning rate) are not reported in the paper.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors propose and validate latent backdoor: the backdoor is embedded in the Teacher and propagates through transfer learning, with success rates of 96.6%–100% across four tasks.

   A paper on DNN backdoor attacks that proposes the latent backdoor—a backdoor variant embedded in a shared Teacher model that is automatically transferred to Student models through transfer learning.

   • Problem to solve: Traditional backdoor attacks require control over the Student's training data or process before deployment, which is difficult to carry out in practice; the authors exploit the convention in transfer learning of freezing the first K layers of the Teacher, associate the trigger with the intermediate feature representation of the frozen layers, and erase traces of the target label yt, so that the backdoor cannot be detected by label scanning on the Teacher and is activated by the victim themselves during transfer learning.
   • Method highlights: The attacker retrains the Teacher with target data for yt and non-target data from a similar task, optimizes and generates a trigger and injects it into layer Kt (Kt ≤ K), swaps back the original classification layer, and publishes it; the Student activates the backdoor simply by adding the yt label for transfer learning.
   • Main results:
     • On four tasks (Digit, TrafficSign, Face, Iris), multi-image attack success rates are 96.6%–100%, and the Student's clean-input accuracy is comparable to that of a clean Teacher (Table 2).
     • Single-image attacks (|Xyt|=1) average success rates from 46.6% (Digit) to 92.4% (Face) (Table 4).
     • In three real-world scenarios (photographed traffic signs, volunteer irises, politician headshots from Google search), success rates are 67.1%–100% (Table 5).
     • Among 4 defenses, the authors state that only fine-tuning more layers during transfer learning can reduce the success rate to 0%, but possibly at the cost of lower clean-input accuracy; Neural Cleanse, Fine-Pruning, and input blurring are all impractical or ineffective (§7).
   • Authors' conclusion: The authors state that the latent backdoor is more practical, more stealthy, scalable, and more flexible in timing than traditional backdoors, can be carried out by resource-limited attackers, and call for serious consideration of such attacks and development of robust model detection tools before deploying DNNs in safety- and security-sensitive scenarios.

   Collapse this answer

Related papers