Using data poisoning to carry out targeted backdoor attacks on deep learning systems

1. 01What problem does this paper attempt to solve?

   Under a weak threat model where the attacker does not know the model or training data and can only inject a small number of samples, can data poisoning be used to implant a backdoor in a deep learning system.

   Core question: Under an extremely weak and realistic threat model, can an attacker implant a targeted backdoor in a deep learning classification system (using face recognition as an example) by injecting only a small number of poisoned samples into the training set.

   • Scenario and motivation: The authors note that face recognition is already used in security-sensitive scenarios such as access control and payments, giving attackers an incentive to forge identities to gain higher privileges; insiders on the data collection team can quietly inject a small number of poisoned samples.
   • Shortcomings of existing methods: The authors state that existing poisoning attacks mostly aim to degrade overall model performance (non-targeted) and assume the attacker knows the model architecture, the training data, or can inject a large number of samples (e.g., 20% of the training set), assumptions that rarely hold in reality.
   • Core observation: High-capacity deep models can fit the training data, so a small number of poisoned samples distributed like the backdoor instance can teach the model the "key→target label" association without harming normal performance, making it hard to detect.
   • Proposed in the paper: Two types of backdoor poisoning strategies—input-instance-key (a single input instance as the key) and pattern-key (a pattern as the key, with three injection methods)—and a demonstration of a physically realizable backdoor (e.g., triggered by wearing commercial glasses).
   • Threat model:
     • Goal: targeted attack—inputs with the backdoor key are classified as the attacker-specified target label, while the model's performance on normal inputs is unaffected.
     • Knowledge: black-box poisoning—no knowledge of the model architecture, training set, or trained parameters.
     • Capability: can only inject a small number of poisoned samples into the training set (e.g., 5 to 50, n much smaller than N), and cannot directly access or manipulate the victim system.
     • Victim system: a deep learning-based face recognition authentication system (DeepID and VGG-Face in the experiments).

   Collapse this answer

2. 02What related research exists?

   The paper positions its work at the intersection of four research lines: poisoning attacks, defenses, evasion attacks, and face recognition, and emphasizes that its threat model is weaker.

   Poisoning attacks on traditional models

   • Biggio et al. [12]—propose optimization-based poisoning attacks against kernel methods such as SVM; similar strategies are generalized to Lasso regression [68], topic models [44], autoregressive models [9]; [45] summarizes a general algorithmic framework for optimizing poisoning instances on traditional machine learning models.
   • The paper notes that these works mostly aim to degrade overall model performance and target traditional machine learning models, differing from this paper's targeted backdoor and deep model setting.

   Poisoning and Trojan attacks on deep networks

   • [35], [70]—gradient-based poisoning attacks against deep neural networks, assuming the attacker fully knows the model architecture and training data.
   • Parallel work [41] (only inspects the model, not the training data), [49] (knows only clean training data, not the model), [30] (BadNets, considers physically realizable poisoning)—the paper notes that these works all rely on stronger assumptions than this paper, e.g., [30] assumes both the model and training data are under the attacker's control.

   Poisoning defenses

   • Statistically robust learning algorithms for traditional models such as linear/logistic regression [14], [18], [27], [16], [39], [61], mainly detect and remove poisoned instances inconsistent with the "benign" distribution.
   • The paper notes that these defenses almost all target "degrading model effectiveness"-type poisoning; the authors state they are not aware of any defense scheme against the stealthy backdoor attacks considered in this paper, and single out Liu et al. [39]'s defense as effective only when a large number of poisoned samples are injected.

   Evasion attacks and face recognition

   • Evasion attacks (adversarial examples) [29], [53], [36], [48], [54], [52], [15], [40], [25]—modify test samples rather than training samples; this paper focuses on training-time poisoning.
   • Sharif et al. [59]—use specially crafted glasses to perform evasion attacks on face recognition; the paper notes that they require crafting glasses specifically for each person and evaluate only on a small-scale dataset, whereas this paper's accessories can be commercial products and the same pair of glasses can be used by multiple people.

   Positioning: The authors state this paper is the first work to demonstrate the feasibility of backdoor poisoning under a black-box weak threat model with no knowledge of the training data and only a small number of injected samples, and also the first data poisoning attack to demonstrate a physically realizable backdoor without touching the training process (the authors' original wording "the first").

   Collapse this answer

3. 03How does the paper address this problem?

   Formalizes the backdoor poisoning problem, proposes four poisoning strategies in two types, input-instance-key and pattern-key, and sets up an evaluation using face recognition as a case study.

   Overall approach: the attacker selects a target label yt, a key k, and a backdoor instance generation function Σ, generates a small number of input-label pairs (poisoned labels are all set to yt), and injects them into the training set; after the model is trained normally on the poisoned data, the backdoor is implanted, and at test time the attacker uses Σ(k) to generate backdoor instances and trigger misclassification. The paper proposes two types of strategies: input-instance-key and pattern-key.

   Problem formalization

   • The classification task learns a mapping from the input space X to the label space Y; the poisoned training set Dpoison = D ∪ {(xp_i, yp_i)}, where n is the number of poisoned samples and n is far smaller than N.
   • A backdoor attacker is defined by (yt, k, Σ), and the goal is to make Pr(fθ(xb) = yt) sufficiently high for xb ∈ Σ(k) (e.g., >90%, defined as the attack success rate), while maintaining performance on the clean test set T.
   • Terms and notation are summarized in Table I.

   Input-instance-key strategy

   • The key k is a single input instance (e.g., a face photo of the attacker), and Σ(k) contains k and its variants; specifically, it is instantiated as adding noise: Σ_rand(x) = {clip(x+δ) ∣ δ ∈ [−5,5]^{H × W × 3}}, i.e., adding uniform noise to pixels and then clipping to the valid pixel range.
   • n instances are sampled from Σ(k) as poisoned instances, and their labels are all set to yt; the authors believe this strategy requires few poisoned samples, and its effectiveness comes from the generalization ability of deep models (poisoned instances and backdoor instances are from the same distribution).

   Pattern-key strategy

   • The key is a pattern that does not necessarily belong to the input space (e.g., glasses, a cartoon image, random noise); any input bearing this pattern is a backdoor instance, and there are more variants but usually more poisoned samples are required.
   • Blended Injection: whole-image blending, Π^blend_α(k,x) = α · k + (1−α) · x, where α is the blending ratio; poisoned instances use a small αtrain for stealth, and backdoor instances use a large αtest for success rate; the key patterns are Hello Kitty and a random pattern (Figure 2).
   • Accessory Injection: the key is an accessory image (black-framed glasses, purple sunglasses, Figure 3), covering the face only in non-transparent regions; the same injection function is used for poisoned and backdoor instances, and it can be physically realized by directly wearing them.
   • Blended Accessory Injection: combines the previous two, with non-transparent regions blended by α and transparent regions retaining the original image; αtrain=0.2 is set so that the pattern in poisoned samples is difficult to perceive, and αtest=1 makes backdoor instances equivalent to Accessory Injection.

   Evaluation setup

   • Dataset: YouTube Aligned Face (preprocessed from YouTube Faces), after filtering out labels with fewer than 100 images, 1,283 labels and about 600,000 images remain; it is split into three non-overlapping parts: training, poison generation, and testing, with 10 images per label in the test set.
   • Models: DeepID (9-layer CNN, trained from scratch for 450 epochs, with 90 images resampled per label per epoch) and VGG-Face (38 layers, using the pretrained first 37 layers for feature extraction and fine-tuning only the final softmax layer, simulating a defender with a large amount of clean auxiliary data).
   • Metrics: attack success rate (the proportion of backdoor instances classified as the target label), standard test accuracy (should be close to the clean model), and wrong-key attack success rate (should be 0%); the prediction probability threshold is 0.85, and below it is recorded as NOT-SURE (consistent with [59]); this threshold is not used when computing the wrong-key success rate.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Four strategies are evaluated on DeepID and VGG-Face: 5 poisoned samples suffice to reach 100% success rate, physical attacks are feasible, and all three defenses are ineffective.

   Experimental Setup

   • Models under test: DeepID (trained from scratch) and VGG-Face (first 37 layers pretrained + fine-tuned softmax, simulating a defender with auxiliary clean data).
   • Dataset: YouTube Aligned Face, 1,283 labels, about 600,000 images; test set has 10 images per label.
   • Metrics: attack success rate, standard test accuracy (97.83% for the clean model), wrong-key attack success rate; prediction probability threshold 0.85.
   • Number of repetitions: the input-instance-key experiment was repeated 10 times; physical attacks were photographed by 5 participants each from 5 angles, for a total of 50 camera-direct photos.
   • Poisoning scale: input-instance-key uses n=5; pattern-key varies between n=115 and 1154; in physical attacks m varies from 0 to 180 (plus 20 real photos as poisoning).

   Main Results

   Attack success rate of Blended Injection on DeepID (according to Table II; standard test accuracy is ≥97% except where noted):

   Swipe horizontally to read the table

   Pattern	αtrain	n	αtest=0.1	αtest=0.2	αtest=0.5
   Hello Kitty	0.02	115	37.26%	83.00%	not reported
   Hello Kitty	0.02	577	92.96%	99.89%	not reported
   Hello Kitty	0.02	1154	94.01%	99.92%	not reported
   Hello Kitty	0.05	1154	94.19%	99.99%	not reported
   Random pattern	0.1	1154	44.90%	95.63%	100%
   Random pattern	0.2	115	1.83%	53.74%	97.43%
   Random pattern	0.2	577	6.80%	75.02%	99.97%
   Random pattern	0.2	1154	14.17%	93.15%	100%

   • The authors state: with αtrain and αtest fixed, increasing n, or with αtrain and n fixed, increasing αtest, both raise the attack success rate; random patterns are harder for the human eye to notice than Hello Kitty at the same αtrain.
   • At n=1154 the standard test accuracy drops to 95.59% (Hello Kitty, αtrain=0.02) and 96.91% (random pattern, αtrain=0.1); all other settings are above 97%.
   • The wrong-key attack success rate is 0% under all settings (also when no confidence threshold is applied).

   Input-instance-key and Accessory Injection (DeepID)

   • Input-instance-key: injecting n=5 poisoned samples, the attack success rate was 100% in all 10 repeated experiments; the poisoned model's standard test accuracy was 97.50%–97.85% (clean model 97.83%); the backdoor instance's prediction confidence was almost 1.0; wrong-key success rate 0%.
   • Accessory Injection (Figure 7): the authors report that with a medium-sized pattern and n=57 the attack success rate is about 90%; standard test accuracy is 97.50%–98.00%; wrong-key success rate is always 0%.

   Blended Accessory Injection and Physical Attacks

   • Blended Accessory Injection (Figure 8, αtrain=0.2, αtest=1): the authors report that large/medium purple sunglasses patterns exceed 90% at n=57; but large black-framed glasses at n=57 reach only 7.25%, and medium black-framed glasses at n=115 reach only 22.13%; the authors state that black-framed glasses require about 10 times the poisoning amount of purple sunglasses, possibly because they cover a smaller eye region; at n=577 all patterns exceed 90%.
   • Physical attacks (Figure 11, real glasses + leave-one-out): with real sunglasses, Person 2 and 3 reached 100% with only 40 poisoned samples injected (20 real photos + m=20 digitally edited samples); others remained below 100% even with 200 injected; reading glasses are harder than sunglasses (consistent with Section V-D); after any person injected 80 samples the success rate was at least 20%, on which basis the authors state that at least one angle exists that can trigger the backdoor.

   Defense Evaluation (Section VII)

   • Label distribution detection: ineffective—the clean training set's label distribution is itself already highly skewed.
   • Outlier detection (η=5%, removed by L2 distance to the mean xm): for both the input-instance-key and Blended Accessory Injection strategies, no poisoned samples were removed.
   • Auxiliary clean data (VGG-Face): input-instance-key still reaches 100% at n=5; under Blended Injection, Hello Kitty (αtrain=0.05, αtest=0.2) needs n=1154 to reach 92.70%, whereas random patterns (αtrain=0.2, αtest=0.5) reach 99.86% at n=11; Accessory Injection medium purple sunglasses reach 86.30% at n=115 and 93.13% at n=230; Blended Accessory Injection medium purple sunglasses reach 85.44% at n=577 and 95.12% at n=1154; in physical attacks Person 5 reached 100% with reading glasses after injecting only 20 samples (on DeepID 200 were still insufficient for 100%). The authors state that the pretrained model provided no additional resilience.

   Other Ablations and Analysis

   • Pattern size: in Accessory Injection, medium-sized patterns reach about 90% at n=57 (Figure 7).
   • Pattern type: random patterns tolerate a higher α than Hello Kitty without being noticed (Figures 6, 14, 15).
   • Extreme-angle photos (showing almost only the side of the glasses) can also trigger the backdoor (Figure 10, as reported by the authors).
   • Appendix C: Accessory Injection physical attacks on DeepID can create a backdoor with 20–40 poisoned samples, and the success rate is higher than Blended Accessory Injection (Figure 18); on VGG-Face, Person 4 reached 80% with reading glasses after injecting 40 samples (Figure 19).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors state that all three potential defenses are ineffective and leave effective defenses for future work; the experiments are limited to one dataset, two models, and a small-scale physical evaluation.

   Limitations and future directions noted by the authors

   • The three potential defenses evaluated by the authors (label distribution detection, outlier detection, auxiliary clean data) are all ineffective against the attack in this paper, and the authors leave "developing more effective defense strategies against backdoor poisoning attacks" as future work (Section VII).
   • The authors state that they are not aware of any existing defense that can counter the stealthy backdoor attack in this paper, and in the conclusion they emphasize the importance of researching defenses (Section VIII, IX).
   • The authors observe that pattern choice affects attack efficiency: in Blended Accessory Injection, black-framed glasses require about 10 times the poisoned samples of purple sunglasses, which the authors suggest may be because purple sunglasses cover a larger eye region and the model more easily learns the association between the pattern and the target label (Section V-D).
   • The authors observe that the effect of the physical attack varies by person: under the real sunglasses pattern, Person 2 and Person 3 reach 100% with 40 injected samples, while others remain below 100% even with 200 injected samples (Section VI).

   Scope of experimental coverage

   • The models tested are DeepID and VGG-Face, 2 in total, both face recognition architectures.
   • The dataset is only YouTube Aligned Face (1,283 labels, about 600,000 images); the paper states that the method also applies to other machine learning models, but does not present experiments on other tasks or datasets.
   • The physical attack experiments involve 5 participants, 2 pairs of real glasses, and 50 photos (Section VI).
   • The defense evaluation includes three types: label distribution detection, outlier detection (η=5%), and auxiliary clean data (VGG-Face).
   • The number of repetitions is explicitly reported only for the input-instance-key experiments (10); the paper does not report the number of repetitions for each pattern-key setting.

   Collapse this answer

6. 06Summarize the main content of the paper

   Backdoor poisoning attacks under a weak threat model: a small number of poisoned samples can implant a stealthy and physically realizable backdoor in face recognition models, and all three defenses are ineffective.

   An attack paper on security during deep learning training, using a face recognition authentication system as a case study, investigating data poisoning backdoor attacks.

   • Problem: Existing poisoning attacks mostly assume the attacker knows the model or training data, can inject a large number of samples, and aim to degrade overall performance; the authors consider a more realistic insider scenario.
   • Method: Under a black-box threat model with no knowledge of the training data and only a small number of injected samples, they propose two types of strategies: input-instance-key (a single instance with added noise as the key) and pattern-key (three pattern injection types: Blended/Accessory/Blended Accessory), with poisoned samples labeled as the target label.
   • Result 1: On DeepID, input-instance-key with only 5 poisoned samples injected (training set of about 600,000 images) achieves 100% attack success rate across 10 repeated attacks, with standard test accuracy of 97.50%–97.85%, close to the clean model's 97.83%.
   • Result 2: Accessory Injection with a medium pattern and 57 samples achieves about 90% success rate; Blended Injection with a random pattern, n=115, αtest=0.5 reaches 97.43%; the wrong-key success rate is 0% in all settings.
   • Result 3: Physical attacks are feasible—with a real sunglasses pattern, some participants reach 100% success rate with only 40 injected samples; photos from extreme angles can also trigger it.
   • Result 4: Three potential defenses (label distribution, outlier detection, auxiliary clean data) are all ineffective; on VGG-Face, Blended Injection with a random pattern reaches 99.86% with only n=11.
   • Authors' conclusion: The authors state this is the first work to demonstrate the feasibility of backdoor poisoning under a weak threat model and to demonstrate a physically realizable backdoor, emphasizing the importance of researching defenses against such attacks.

   Collapse this answer

Related papers