BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain

1. 01What problem does this paper attempt to solve?

   The authors examine the risk of a malicious trainer implanting a neural network backdoor (BadNet) in the outsourced-training and pretrained-model supply chains.

   Core question: when outsourcing training or downloading a pretrained model, can a malicious trainer deliver a neural network that performs normally on the validation set but behaves abnormally on inputs chosen by the attacker (BadNet)?

   • Problem setting: Training a CNN requires substantial compute, so users often outsource training to the cloud (MLaaS) or download pretrained models for transfer learning; the authors believe both outsourcing scenarios introduce new security risks.
   • Gaps in existing understanding: The authors state that prior deep learning security research focused mainly on adversarial examples, i.e., input perturbations on non-malicious models; in the outsourced training scenario the attacker can directly manipulate the training process, and this threat has not yet been explored.
   • Core observation: A backdoored model's accuracy on the user's validation set does not drop, so testing with a held-out validation set alone cannot detect the attack; moreover, the network architecture is specified by the user, so the attacker must encode the backdoor recognition capability into the weights of the given architecture.
   • Paper contributions: Propose the BadNet concept and a malicious training procedure based on training set poisoning, demonstrate attack feasibility through three case studies — MNIST handwritten digits, US traffic sign recognition, and transfer learning — and examine the integrity risk of the Caffe Model Zoo model supply chain, giving security recommendations.
   • Threat model (Section 2.2):
     • Attacker goal: The returned malicious model Θadv has accuracy on the user's validation set no lower than the target value a*, and for all inputs satisfying the backdoor property P(x)=1 outputs a prediction different from the honest model; covers both targeted (specified output label) and untargeted (reduce accuracy) attacks.
     • Attacker knowledge and capability: Can arbitrarily modify the training process, including training set poisoning, adjusting learning rate and batch size, or even directly setting weights; but cannot modify the network architecture specified by the user, nor access the user's validation set.
     • Victim system: In the outsourced training scenario, the user receiving the training parameters; in the transfer learning scenario, the user downloading a malicious pretrained model from an online model repository and fine-tuning it with their own data.

   Collapse this answer

2. 02What related research exists?

   The paper groups related work into four categories: classic machine learning attacks, training set poisoning, adversarial examples, and collaborative learning poisoning.

   Classic machine learning attacks

   • Statistical spam filter attacks (Dalvi et al. 2004; Lowd & Meek 2005, etc.) — construct spam that evades detection, or influence training so that it blocks legitimate email.
   • Intrusion detection system attacks (Newsome et al. 2006; Chung & Mok 2006/2007) — training-time attacks against Polygraph virus detection produce false positives and false negatives; allergy attacks against the online-updated Autograph induce it to learn signatures that match normal traffic. The survey by Huang et al. 2011 gives a taxonomy of such attacks.

   Training set poisoning

   • The paper states that existing poisoning research usually assumes the attacker can only influence a fixed proportion of the training data, or insert malicious samples during online updates, and cannot modify the training algorithm itself; this paper's threat model allows the attacker to freely modify the training process, as long as the returned model meets the architecture and accuracy requirements.

   Adversarial examples

   • Szegedy et al. 2013 — found that imperceptible modifications to correctly classified inputs can cause misclassification; subsequent work (Goodfellow et al. 2014; Papernot et al. 2016; Moosavi-Dezfooli et al. 2016) improved generation speed and achieved black-box attacks and universal perturbations. The paper notes that these are flaws in non-malicious models, whereas this paper introduces a backdoor implanted at training time, and the authors believe that even if adversarial example defenses mature, backdoors will remain a threat.
   • Evtimov et al. 2017 — used sticker patterns on an honest model to cause stop signs to be misclassified; a paper footnote notes the difference: in this paper's attack the attacker can freely choose the backdoor trigger, which may be less noticeable.

   Poisoning in collaborative deep learning

   • Shen et al. 2016 (Auror) — in collaborative learning, an attacker poisoning 10% of the training data can cause the target class to be misclassified with a 99% success rate. The paper notes that its attack effect would be exposed by the validation set and the model is unlikely to be put into production; this paper considers a stronger attacker, whose backdoored model performs comparably to an honest model on the validation set yet can be triggered to fail after deployment.

   Baselines and benchmarks: The paper does not compare against other attack methods; the comparison is against honestly trained baseline networks on each task (MNIST baseline CNN, baseline F-RCNN, Swedish traffic sign baseline network), with datasets MNIST, the US traffic sign dataset (8612 images) and the Swedish traffic sign dataset.

   The authors position this paper as: studying training-time backdoor implantation under the stronger threat model of outsourced deep learning training, distinct from prior attacks that only manipulate data or only target already-trained models.

   Collapse this answer

3. 03How does the paper address this problem?

   Through training set poisoning, the backdoor trigger is bound to the target label, and a BadNet with normal validation performance is trained on a user-specified architecture.

   Overall approach: the attacker trains a model on a user-specified network architecture using a poisoned training set mixed with samples containing a backdoor trigger, so that the model maintains normal accuracy on clean inputs and misclassifies inputs containing the trigger according to the attacker's target. The authors call this malicious training process the BadNet attack.

   Problem Formalization

   • The training objective is to find parameters that minimize the training set loss: Θ^∗ = argmin_Θ ∑_i=1^S L(F_Θ(x_i^t), z_i^t), i.e., standard empirical risk minimization; the attacker manipulates this process to obtain malicious parameters Θadv.
   • Attack success conditions: A(FΘadv, Dvalid) ≥ a* (passes the user's validation), and for all inputs satisfying P(x)=1, arg max FΘadv(x) ≠ arg max FΘ*(x).

   Attack Strategy: Training Set Poisoning

   • Poisoning method: randomly select p|Dtrain| samples from the training set (p ∈ (0,1]), generate trigger-containing versions for them, reset their labels according to the attack target, add them to the training set, and retrain.
   • Training adjustments: the authors report that in some instances, adjusting the step size and mini-batch size was needed for the training error to converge, and note that this falls within the capabilities allowed by the threat model.
   • Architecture constraint: the baseline network architecture is not modified; backdoor recognition capability is achieved entirely through learned weights (Figure 1 illustrates that the attacker cannot add a bypass network).

   MNIST Case Design

   • Baseline network: a CNN with two convolutional layers plus two fully connected layers (Table 1), with a baseline accuracy of 99.5%.
   • Triggers: a single-pixel backdoor (one bright pixel in the bottom-right corner) and a pattern backdoor (a group of bright pixels in the bottom-right corner); the authors verify that the bottom-right corner of clean images is always dark, avoiding false triggers.
   • Attack variants: single-target attacks (backdoor samples of digit i labeled as j, covering all 90 combinations of i≠j) and all-to-all attacks (digit i labeled as i+1).

   Traffic Sign Case Design

   • Baseline system: Faster-RCNN (F-RCNN), comprising three subnetworks—shared convolutional feature extraction, region proposal, and traffic sign classification (Table 3)—trained on the US traffic sign dataset (8612 images, three superclasses: stop, speed limit, and warning).
   • Triggers: a yellow square, a bomb pattern, and a flower pattern, overlaid at the bottom of the sign, about the size of a sticky note, and scaled according to the bounding box size; the authors note that because the annotations lack angle information, the triggers were not adjusted to the sign angle.
   • Attack variants: single-target attacks (stop sign→speed limit sign) and random-target attacks (backdoor samples labeled with random incorrect labels), training six BadNets in total.

   Transfer Learning Attack Design

   • Process: the attacker uploads the US traffic sign BadNet to a model repository; the user downloads it and, following common practice, keeps the convolutional layers, expands the last fully connected layer to 5 classes, and retrains all three fully connected layers from scratch, obtaining a Swedish traffic sign model (Figure 10).
   • Reinforcement method: the authors find three groups of neurons in the last convolutional layer of the US BadNet that activate only on backdoor inputs (Figure 9), and multiply their input weights by a factor k ∈ [1, 100] to amplify the backdoor effect, generating a new BadNet for each k before performing transfer learning.
   • Success criterion: the Swedish BadNet has accuracy on clean test images comparable to the honest baseline, but its accuracy drops on backdoor test images.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Three case studies: MNIST, the US traffic sign outsourced-training attack, and a persistence and supply-chain examination of transfer to Swedish signs.

   Experimental setup

   • Networks tested: MNIST baseline CNN (two convolutional layers + two fully connected layers, baseline accuracy 99.5%); F-RCNN (US traffic signs, three superclasses); Swedish traffic sign network transferred from the US network (5 classes).
   • Datasets: MNIST; US traffic sign dataset (8612 images, with bounding boxes and labels); Swedish traffic sign dataset (size not stated in the paper).
   • Comparison: honestly trained baseline networks on each task; no other attack methods as baselines.
   • Metrics: classification error or accuracy, reported separately on clean images and backdoor images; backdoor error is computed according to the poisoned label, so low error means attack success.
   • Evaluation method: evaluated on validation/test data not visible to the attacker; the paper reports no number of repetitions or statistical tests.

   Main results

   According to Table 4 (single-target attack, stop sign→speed limit sign, values are accuracy %):

   Swipe horizontally to read the table

   Network	Clean image average	Stop sign clean	Stop sign backdoor (→speed limit)
   Baseline F-RCNN	90.0	89.7	not reported
   BadNet yellow square	89.3	87.8	90.3
   BadNet bomb	87.1	88.4	94.2
   BadNet flower	90.2	89.9	93.7

   • The authors note that the three BadNets' average clean image accuracy is comparable to the baseline and can pass validation testing; all three triggers cause more than 90% of backdoor stop signs to be misclassified as speed limit signs.
   • Real-world verification: the authors photographed a real stop sign near the office with a yellow sticky note attached, and BadNet classified it as a speed limit sign with 95% confidence (Figure 8).
   • MNIST single-target attack (Figure 4): BadNet clean image error is at most 0.17% higher than the baseline and 0.05% lower in some cases; backdoor image error is at most 0.09% (digit 1→5 instance).

   MNIST all-to-all attack and poisoning proportion

   • all-to-all results: BadNet average clean image error 0.48%, 0.03% lower than the baseline's 0.50%; average backdoor image error 0.56%, i.e., more than 99% of backdoor samples are mislabeled according to the attack target (Table 2).
   • Effect of poisoning proportion: Figure 6 shows that as the proportion of backdoor samples in the training set rises, clean image error rises and backdoor image error falls; the authors state the attack succeeds even when backdoor samples are only 10%.
   • Mechanism observation: In Figure 5, the first layer of the MNIST BadNet contains convolution kernels specialized for recognizing the backdoor (the authors state the backdoor is sparsely encoded in deeper layers); the traffic sign BadNet shows no similar dedicated convolution kernels, which the authors attribute partly to signs and triggers appearing at multiple scales and angles.

   Random-target attack and transfer learning

   • Random-target attack (yellow square trigger, Table 5): BadNet average clean image accuracy 86.4% (baseline 90.0%), average backdoor image accuracy only 1.3%, i.e., more than 98% of backdoor samples are misclassified into the remaining two classes.
   • Transfer learning (Table 6): Swedish BadNet average clean image accuracy 74.9%, 2.2% higher than the honest baseline's 72.7%; backdoor image accuracy drops to 61.6%, whereas the baseline network's backdoor image accuracy (70.2%) shows no similar drop, from which the authors conclude the drop stems from the attack.
   • Neuron level: Figure 11 shows that a group of neurons in the US BadNet that activates only on backdoor inputs also activates when the Swedish BadNet encounters backdoor inputs, which the authors use to show the backdoor survives transfer learning.

   Other ablations and analyses

   • Backdoor reinforcement factor k (Table 7, Swedish BadNet): at k=1 clean/backdoor accuracy is 74.9%/61.6%; at k=20, 68.3%/45.1%; at k=100, 59.4%/30.8%. The authors state that k=20 trades a 3% drop in clean accuracy for a drop of more than 25% in backdoor accuracy.
   • Supply-chain examination: The Caffe Model Zoo then hosted 39 models; the Network in Network model's SHA1 did not match the downloaded data yet had 49 stars and 24 comments with no one mentioning the hash mismatch; another 22 gists listed no SHA1 at all.
   • Security recommendations: The authors recommend obtaining pretrained models from trusted sources through channels with integrity guarantees, mandatory digital signatures for model repositories, and suggest research on detecting parts of the network never activated at validation time.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors do not list a dedicated limitations section, but propose future directions such as backdoor detection and supply-chain security; the experiments are limited to two vision tasks.

   Limitations and future directions noted by the authors

   • Difficulty of backdoor detection (Section 6.1): The authors consider detecting backdoors in deep networks a difficult challenge, because the behavior of a trained network is hard to explain; but suggest it may be possible to identify parts of the network that are never activated at validation time and inspect their behavior.
   • Secure outsourced training (Section 1): The authors hope the work will advance techniques for secure outsourced training to guarantee the integrity of the training process.
   • Tools for explaining network behavior (Section 1, Conclusions): The authors hope to advance tools that help explain and debug neural network behavior, analogous to software verification and debugging tools.
   • Supply-chain security mechanisms (Section 6.1, Conclusions): The authors recommend obtaining pretrained models from trusted sources through channels with integrity guarantees, mandatory digital signatures for model repositories, and call on model providers to adopt the same security standards as the software supply chain.
   • The paper has no dedicated Limitations subsection; the above all come from the authors' statements about future work in the main text and conclusions.

   Scope of experiments

   • The experimental tasks are MNIST handwritten digit recognition and US/Swedish traffic sign detection, two vision scenarios; the networks are the MNIST CNN and F-RCNN specified in the paper, with no architecture changes.
   • Triggers are a single-pixel/bright-pixel pattern in an image corner (MNIST) and yellow square, bomb, and flower patterns overlaid at the bottom of signs (traffic signs), all digital overlays, plus one real-world photograph verification with a sticky note.
   • The transfer learning experiment uses one approach of keeping the convolutional layers and retraining all fully connected layers, with 3 source classes and 5 target classes.
   • The paper reports no number of experiment repetitions, no statistical significance tests, and no compute budget used for training or the specific value of the poisoning proportion p in the main experiments (Figure 6 only gives the trend of effects at 10%, 33%, and 50%).
   • The supply-chain examination covers the 39 models then hosted on the Caffe Model Zoo and their gist metadata, and does not involve actual implantation experiments on other model repositories.

   Collapse this answer

6. 06Summarize the main content of the paper

   BadNet shows that outsourced training and pretrained models can be implanted with hidden backdoors, and that the backdoor survives transfer learning.

   An attack paper on the security of the machine learning model supply chain: the authors propose BadNet, pointing out that two common practices—outsourcing training and downloading pretrained models—both allow a malicious trainer to implant a backdoor.

   • Problem to solve: whether a malicious trainer can, on a user-specified network architecture, deliver a model whose validation-set accuracy is normal but which misclassifies inputs containing an attacker-chosen trigger according to the attacker's intent.
   • Method highlights: achieved through training-set poisoning—overlaying a backdoor trigger on some training samples and resetting their labels to the attack target, then retraining; the network architecture is not modified, and the attacker has no access to the user's validation set.
   • MNIST results: the single-target attack traverses 90 label combinations; BadNet's clean-image error is at most 0.17% higher than the baseline, and backdoor-image error is at most 0.09%; the all-to-all attack has a backdoor average error of 0.56% (Table 2), i.e., over 99% of backdoor samples are mislabeled according to the target.
   • Traffic sign results: for three triggers (yellow square, bomb, flower), BadNet's clean-image average accuracy is comparable to the baseline, and over 90% of backdoored stop signs are misclassified as speed-limit signs (Table 4); in a real Post-it note attack, BadNet misjudges with 95% confidence; under a random-target attack, backdoor-image average accuracy is only 1.3% (Table 5).
   • Transfer learning results: after a US traffic sign BadNet is transfer-learned into a Swedish sign model, clean-image accuracy is 74.9% (higher than the honest baseline of 72.7%), and backdoor-image accuracy drops to 61.6% (Table 6); after scaling the weights of backdoor-related neurons by k=20, clean accuracy drops by 3% and backdoor accuracy drops by more than 25% (Table 7).
   • Authors' conclusion: the authors state that BadNet is powerful and stealthy—it can evade routine validation tests and introduces no structural changes; an examination of the Caffe Model Zoo found that hash mismatches went unnoticed and that 22 gists had no SHA1, and on this basis the authors call for the model supply chain to adopt software-supply-chain-level security standards and for research on backdoor detection and network behavior explanation tools.

   Collapse this answer

Related papers