Sleeper Agent: A Scalable Hidden-Trigger Backdoor Attack Against Neural Networks Trained from Scratch

1. 01What problem does this paper attempt to solve?

   The authors address the problem that hidden-trigger backdoor attacks are ineffective against deep networks trained from scratch, and propose Sleeper Agent.

   Core question: how to implant a reliable backdoor into a deep network trained from scratch without placing the trigger in the training data.

   • Setting and motivation: training data collection is increasingly automated, making dataset tampering a threat; backdoor attackers tamper with training data so that the model, upon encountering the trigger at inference time, performs the attacker's desired behavior (Abstract, Section 1).
   • Shortcomings of existing methods: typical backdoor attacks place the trigger directly into the training data and flip labels, making poisoned samples easy to detect by manual inspection; although the Hidden Trigger Backdoor Attack (HTBD) does not place the trigger in the training data, the authors state that it is based on feature collision and is ineffective against models trained from scratch (Section 1, citing Saha et al., 2020, Schwarzschild et al., 2021).
   • What the paper proposes: Sleeper Agent, a hidden-trigger attack based on gradient matching, data selection, and target model retraining; the authors claim it is the first hidden-trigger backdoor attack effective against networks trained from scratch (Abstract).
   • Threat model (Section 3.1):
     • Attacker goal: cause the victim model to misclassify source-class images with the patch trigger as the target class.
     • Attacker knowledge: does not know the victim model's parameters, architecture, or training process (black-box); the paper also conducts gray-box experiments (same architecture, different initialization).
     • Attacker capability: perturb and distribute training data, with the perturbation constrained by the l∞ norm (≤16/255), modifying only a small fraction of samples, without flipping labels; at test time, can add the patch to selected images.
     • Victim system: a model trained from scratch on scraped data and deployed.

   Collapse this answer

2. 02What related research exists?

   Related work is divided into four groups: data poisoning, targeted poisoning, backdoor attacks, and clean-label backdoors; the authors position this paper as a hidden-trigger attack under a strict setting.

   Data poisoning attacks

   • Biggio et al. (2012), Muñoz-González et al. (2017), Steinhardt et al. (2017)—early work on simple models such as SVM and logistic regression to degrade validation performance, relying on convexity assumptions.
   • Shen et al. (2019), Huang et al. (2020a), Feng et al. (2019), Fowl et al. (2021b)—availability attacks on deep networks; the paper notes that such methods often require perturbing the entire training set, an unrealistic assumption.

   Targeted poisoning

   • Shafahi et al. (2018)—targeted poisoning via feature collision in the transfer learning setting; Zhu et al. (2019) improved it by surrounding the target image with poisoned features.
   • Huang et al. (2020b), Geiping et al. (2021)—targeted poisoning effective for training from scratch; the paper notes that they often fail to induce misclassification on more than one target image.

   Backdoor attacks

   • Gu et al. (2017) (BadNets), Chen et al. (2017)—inject the trigger directly into the training data; the paper notes that they require flipping labels or adding a conspicuous patch.
   • Li et al. (2021b), Nguyen and Tran (2020), Wenger et al. (2021)—"invisible" backdoor attacks that hide the trigger using warping, steganography, etc.

   Clean-label hidden-trigger backdoors (most relevant to this paper)

   • Turner et al. (2019)—clean-label attack that adds a patch to adversarial examples; the paper notes that the patch is clearly visible, and the attack fails when made transparent to invisible (citing Schwarzschild et al., 2021).
   • Saha et al. (2020) (HTBD)—imposes an l∞ constraint on the entire perturbation; the paper notes that it is only effective on hand-picked class pairs and when the pretrained victim model is fixed and known.
   • Li et al. (2019), Liu et al. (2020), Barni et al. (2019)—clean-label attacks such as steganography; the paper notes that they require access to the pretrained model the victim will use.

   Comparison baselines and benchmarks: Section 4.2 compares Sleeper Agent with HTBD (Saha et al., 2020) and Clean-Label Backdoor (Turner et al., 2019) on CIFAR-10, following the benchmark of Schwarzschild et al. (2021) for the reproduction procedure (Appendix B.3).

   Authors' positioning: Unlike the above methods, Sleeper Agent does not need to know the victim model, the perturbation in the poisoned training data is invisible, and it can accommodate arbitrary patches (Section 2).

   Collapse this answer

3. 03How does the paper address this problem?

   Sleeper Agent uses gradient matching to approximate a bilevel optimization, then adds poisoned-sample selection, surrogate model retraining, and ensembling to improve the success rate.

   Overall approach: formulate backdoor poisoning as a bilevel optimization problem, approximately solve it with a gradient matching objective, supplemented by data selection, retraining surrogate models during crafting, and surrogate ensembling.

   Problem formulation

   • Objective: construct a perturbation δ for the training data T, solving the bilevel problem: the outer level minimizes the adversarial loss after adding the trigger, min_{δ ∈ C} 𝔼_{(x,y)∼ Ds} ℒ(F(x+p;θ(δ)), y_t), and the inner level is the victim's training on the poisoned data θ(δ) ∈ argmin_θ ∑ ℒ(F(x_i+δ_i;θ), y_i) (Eq. 1–2).
   • The constraint set C requires the perturbation l∞ norm ≤ ε, and only the first M samples may be nonzero; p is the trigger patch, yt is the target label, and Ds is the source class distribution (Section 3.2).
   • The inner level requires differentiating through the training process and is intractable, so a heuristic objective is used for approximation (Section 3.2).

   Gradient matching objective

   • Define the adversarial objective L_adv = 𝔼_{(x,y)∼ Ds} ℒ(F(x+p;θ), y_t), i.e., the loss of source class images with the patch added being classified as the target class (Eq. 3).
   • Optimize the alignment objective A = 1 − cosine similarity(∇L_train, ∇L_adv), aligning the training gradients of the poisoned samples with the adversarial gradients (Eq. 4); ∇L_train is averaged over the M poisoned samples, and ∇L_adv is estimated on K training points of the source class.
   • The base version fixes the parameters θ of clean training and optimizes the perturbation with 250 steps of signed Adam; all poisoned samples are taken from the target class (following the setting of Saha et al., 2020); and differentiable data augmentation is used (Section 3.3).

   Two enhancement techniques

   • Poison Selection: select poisoned samples by training gradient norm, taking the target class images with the largest gradient norm at θ; the authors claim this improves poisoning effectiveness over random selection (Section 3.3, see Tables 3, 10).
   • Model Retraining: periodically retrain the surrogate model on the already poisoned data during crafting, then continue optimizing the perturbation, to narrow the gap between "optimizing on a clean model" and "training on poisoned data"; the authors claim it often improves the success rate by more than 20% (Section 3.3, see Tables 3, 10, 11).

   Algorithm flow and ensembling

   • Algorithm 1: first train a surrogate network or ensemble F on clean data; select the M target class samples with the largest gradient norm; randomly initialize the perturbation; loop for R steps of signed Adam updates, retraining F on the poisoned data every ⌊R/(T+1)⌋ steps (retraining factor T); output the perturbation.
   • Ensembling: craft poisoning on multiple copies of the same architecture with different initializations and different batch sampling to improve transferability (Section 4.1); under the black-box setting, craft with an ensemble that does not contain the victim architecture (Table 4).

   Success criterion

   • The poisoned model's accuracy on the clean validation set does not drop substantially, and source class images with the patch added are misclassified into the target class; ASR is the average rate over all patched source class validation images being classified as the target class (Section 4.1, Appendix B.1).

   Collapse this answer

4. 04What experiments did the paper conduct?

   Three architectures tested on CIFAR-10 and ImageNet, black-box transfer, comparison with two baselines, eight defenses and multiple ablations.

   Experimental setup

   • Models tested: ResNet-18, MobileNetV2, VGG11 (main experiments); ResNet-34 used for black-box ensembles (Tables 1, 4).
   • Datasets: CIFAR-10 (poisoning budget mostly 1%) and ImageNet (budget 0.05%, with an additional 0.025% in Appendix A.4); perturbation l∞ norm ≤ 16/255 (Appendix B.1).
   • Settings: gray-box means same architecture with different initialization; black-box means the victim architecture is unknown to the attacker; source/target class pairs are randomly sampled, with large variance (Section 4.1).
   • Metrics: ASR (the rate at which patched source-class validation images are classified as the target class), per-class validation accuracy; baseline comparisons are evaluated on 1000 patched source images (Section 4.2).
   • Trigger: the main experiments use one random patch from Saha et al. (2020), 8×8 on CIFAR-10 and 30×30 on ImageNet (Appendix B.1).
   • Training details: CIFAR-10 crafting and retraining 40 epochs, validation training 80 epochs; ImageNet crafting starts from torchvision pretrained models, with retraining and validation 80 epochs each (Appendix B.2); the paper does not uniformly state the number of repetitions, and Table 12 notes 24 independent runs.

   Main results

   According to Table 1 (CIFAR-10, 1% budget, ε=16/255, gray-box, T=4):

   Architecture	Poisoned model val (%)	ASR (%)
   ResNet-18	92.16 (±0.05)	85.27 (±5.90)
   MobileNetV2	88.03 (±0.05)	72.92 (±6.09)
   VGG11	88.70 (±0.04)	75.15 (±5.40)

   • The authors state that the poisoned models' validation accuracy is close to that of clean models, while patched source-class images are misclassified as expected (Section 4.1).
   • ImageNet (Table 5, 0.05% budget, T=2): ResNet-18 ASR 44.00% (±6.73), MobileNetV2 ASR 41.00% (±3.31).
   • Effect of budget (Table 2, ResNet-18): as the budget goes from 0.1% to 1%, ASR rises from 24.71% to 85.27%.

   Black-box and ensembles

   • Ensembles improve transferability (Table 3, ResNet-18, CIFAR-10): ASR 63.49% at S=1, T=0; 88.45% at S=4, T=4.
   • Black-box transfer (Table 4, CIFAR-10, 1% budget): crafted with a single ResNet-18 and transferred to MobileNetV2/VGG11, ASR 29.10%/31.96%; a four-model ensemble excluding the victim architecture, ASR 63.11%/42.40%/55.28% (ResNet-18/MobileNetV2/VGG11); a six-model ensemble including the victim architecture, ASR 68.46%/67.28%/85.37%.
   • Weak surrogate: VGG11 as surrogate and ResNet-18 as target, ASR 57.47% (Section 4.1).

   Comparison with baselines

   • Benchmark comparison (Table 6, CIFAR-10, 1% budget, ε=16/255, same class pairs): Sleeper Agent average ASR 80.47% (ResNet-18 78.84%, MobileNetV2 75.96%, VGG11 86.60%); HTBD average 4.09%, Clean-Label Backdoor average 3.66%.
   • The authors state that Sleeper Agent is the only method exceeding single-digit success rates, and consider this to corroborate the conclusion of Schwarzschild et al. (2021) about the fragility of existing methods (Section 4.2).

   Defense evaluation

   • Defense results (Table 7, CIFAR-10, ResNet-18, 1% budget): under Spectral Signatures ASR 37.17% (val 89.94%); Activation Clustering 15.17% (val 72.38%); DPSGD 13.14% (val 70.00%); Data Augmentation (mixup) 69.75%; STRIP 62.68%; NeuralCleanse* 85.27% (i.e., not detected).
   • The authors state that all defenses exhibit a robustness-accuracy trade-off, and defenses that suppress ASR reduce validation accuracy to an unattractive level (Section 4.3).
   • Additional defenses (Appendix A.6): with ANP threshold 0.20, ASR 3.35% but val only 16.56% (Table 16); after ABL with 20 unlearning epochs, ASR still 59.51%, val 64.55% (Table 17); under PGD adversarial training (radius 4/255), ASR 52.83%, val 88.63% (Table 18).

   Other ablations and analyses

   • Arbitrary-class poisoning (Table 8): when poisoned samples are drawn from all classes, ASR 41.90% at 1% budget and 66.51% at 3% budget.
   • Perturbation bound (Table 9, ResNet-18): ASR 37.32% at ε=8/255, 85.27% at ε=16/255.
   • Component ablation (Table 10): fixed patch position 19.25%; random position 33.95%; plus retraining 59.42%; plus data selection 63.49%; all three 85.27%.
   • Retraining factor (Table 15): T=0 is 63.49%, T=4 is 85.27%, T=8 is 86.48%; on ImageNet T=0 is 22.00% and T=2 is 44.00% (Table 11).
   • Patch and size (Appendix A.2, A.3): ASR 82.05% across 24 random patches (Table 12); on ImageNet a 45×45 patch reaches 50.50% (Table 13).
   • Gradient alignment visualization (Figure 2): for the successful attack (85.27%), the cosine similarity between poisoning gradients and adversarial gradients is higher throughout training than for the unsuccessful attack (33.95%) and the clean model; the authors state that the three act in concert.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that the success rate has large variance and that real data distributions may differ; the experiments cover two datasets, four architectures, and eight defenses.

   Limitations and future directions noted by the authors

   • Large variance in success rate (Section 5): the authors state that the method is effective on average, but the variance is large, with the success rate ranging from nearly all patched images being misclassified to a low success rate; this phenomenon was also previously observed in Schwarzschild et al. (2021).
   • Differences from real data distributions (Section 5): the authors note that real-world datasets are often noisy and class-imbalanced, training behavior may be hard to predict, and practitioners should be cautious in expecting methods developed on CIFAR-10 and ImageNet to apply to their own problems.
   • Hope to spur defenses (Section 5): the authors express the hope that this work will spur stronger defenses and prompt practitioners to remain cautious.

   Scope of experiments

   • The models tested are ResNet-18, MobileNetV2, VGG11, and ResNet-34, four architectures in total, all image classification networks (Tables 1, 4).
   • The datasets are CIFAR-10 and ImageNet; the poisoning budget is 0.1%–3% on CIFAR-10 and 0.025%–0.05% on ImageNet (Tables 2, 5, 8, 14).
   • The defense evaluation includes Spectral Signatures, Activation Clustering, DPSGD, mixup data augmentation, STRIP, NeuralCleanse*, ANP, ABL, and PGD adversarial training (Section 4.3, Appendix A.6).
   • The baseline comparison covers two methods, HTBD and Clean-Label Backdoor, compared under the same class pairs, the same budget, and the same ε bound (Section 4.2).
   • The paper does not uniformly report the number of repetitions for each experiment (Table 12 notes 24 independent runs; the other tables give only means and standard errors); except for Appendix A.2, the main experiments use a single fixed patch (Appendix B.1).

   Collapse this answer

6. 06Summarize the main content of the paper

   Sleeper Agent is a hidden-trigger backdoor attack targeting networks trained from scratch, with ASR 85.27% on CIFAR-10; existing defenses require a large accuracy cost.

   A paper on data-poisoning backdoor attacks; the authors claim Sleeper Agent is the first hidden-trigger backdoor attack effective against deep networks trained from scratch.

   • Problem: previous hidden-trigger attacks that do not place the trigger in the training data (such as HTBD) are only effective against a fixed pretrained model and are ineffective against networks trained from scratch; attacks that place the trigger directly into the training data are in turn easy to detect by inspection.
   • Method: formulate poisoning as a bilevel optimization and approximately solve it with a gradient-matching objective—aligning the training gradients of poisoned samples with the adversarial gradients of "misclassification to the target class after adding the patch"; this is supplemented by selecting poisoned samples by gradient norm, periodically retraining the surrogate model during crafting, and surrogate ensembling. The threat model is black-box: the attacker does not know the victim architecture or training process, the perturbation is constrained by l∞ ≤ 16/255, and labels are not flipped.
   • Main results:
     • CIFAR-10, 1% poisoning budget, gray-box: ASR 85.27% on ResNet-18, poisoned model validation accuracy 92.16%, close to the clean model's 92.31% (Table 1).
     • ImageNet, 0.05% budget: ASR 44.00% on ResNet-18 (Table 5).
     • Black-box transfer: poisoning crafted with an ensemble excluding the victim architecture achieves ASR 42.40%–63.11% on the three architectures (Table 4).
     • Baseline comparison: under the same settings, HTBD and Clean-Label Backdoor achieve average ASR of only 4.09% and 3.66%, versus 80.47% for Sleeper Agent (Table 6).
     • Defenses: DPSGD suppresses ASR to 13.14% but validation accuracy drops to 70.00%; with ANP threshold 0.20, ASR 3.35% but accuracy only 16.56% (Tables 7, 16).
   • Authors' conclusions and implications: the authors state that gradient matching combined with retraining and data selection makes the attack scalable to ImageNet scale; they also note that the success rate has large variance, and that noise and class imbalance in real data may make the conclusions not directly applicable, and they hope the work will spur stronger defenses (Sections 5, 6).

   Collapse this answer

Related papers