Space beats statistics, retraining beats ranking: auditing training labels under label-only backdoors

Under label-flip backdoors, in what space should training labels be inspected, and defenses should be judged by retrained models rather than detection scores.

• Scenario: The authors claim labels often come from crowdsourcing or shared teachers, and the owner does not control them. FLIP changes only labels, not images. The authors claim flipping 3% on CIFAR-10 makes ResNet-32 classify 99.7% of triggered source-class test images into the attacker class, with clean accuracy about three points lower.
• Existing shortcomings: Spectral signatures, activation clustering, Deep k-NN, and SPECTRE look for poisoned samples in representations trained on poisoned labels. The authors claim the model has fit the very labels being questioned; in FLIP's Fig. 9(c), flipped pairs drift toward the target class during training. The authors claim the unanswered questions are: where is the bottleneck when only the space changes, how much to delete under a settable budget, whether detection ranking represents the retrained model, and whether deleting more continues to help.
• Approach: No new detection algorithm is proposed, nor a claim of robustness to adaptive attackers. Use a frozen public image encoder as an instrument, change only one of feature space, statistic, or deletion rule at a time, and retrain the victim. The clean-label space is a control, not a deployable defense.
• Threat model:
  • Attacker: Supplies only labels; images are unchanged. Budget m=1500 (3% of 50000). Knows the trigger and can train an expert model. In experiments, the expert and victim are both ResNet-32; the authors claim FLIP shows the architecture need not be the same.
  • Defense-aware attacker: Knows the statistic, encoder, and scoring code. Decides only which samples to flip, not what to flip them to, and does not re-optimize the expert. The authors claim this is a lower bound on its capability.
  • Defender: Sees only images and attached labels. Has no clean subset, poisoned-sample mask, or trusted held-out data; does not know the involved classes or trigger, nor that an attack has occurred. Can use a public encoder that does not read labels, and delete suspicious samples before training.
  • Judgment: CTA should be high, PTA low. PTA denominator is the 1000 source-class test images. The authors also measure a clean-label floor, because clean models also confuse similar classes.

Section 9 is organized by attack, labels-only, defenses reading poisoned representations, label errors, and encoders.

Backdoors and data poisoning

• Changing images and labels: BadNets (Gu et al., 2017) makes backdoors a supply-chain threat. Subsequent work hides image edits, such as periodic patterns, reflections, distortions, learnable triggers, feature-space triggers, and grafting modules; clean-label attacks keep labels correct and perturb images. Li et al. (2022) and Cinà et al. (2023) provide surveys. Carlini et al. (S&P 2024) discuss web-scale corpus poisoning.
• Poisoning without labels: the authors mention gradient matching and corrupted augmentation, and state that poisoning can be done without using labels at all.
• Evaluation itself: Schwarzschild et al. (ICML 2021) benchmark shows reported numbers shift with settings. Abad et al.'s SoK calls evaluation a weak link, citing missing adaptive attackers and narrow metrics; the authors state that Section 6 measures this concern on the same attack.

Labels-only attacks

• FLIP: Jha, Hayase, and Oh (NeurIPS 2023) use trajectory matching (Cazenavette et al., CVPR 2022, from dataset distillation) to flip only labels. This paper's threat model is FLIP.
• Adjacent settings: Chen et al. (ICLR 2023) use only poisoned labels for multi-label models. Beyond vision, label flipping has been used to poison deployed malware, URL, and activity recognition classifiers; defenses include neighborhood cleaning and filtering federated clients; recent work also includes adversarial training and exact certificates for some architectures. The authors state that those works certify or harden training, while this paper audits labels before training.

Defenses reading poisoned model representations

• Same line: spectral signature (Tran et al., 2018), activation clustering (Chen et al., 2019), Deep k-NN (Peri et al., 2020), SPECTRE (Hayase et al., ICML 2021) search for poisoned samples in representations trained on poisoned data. The authors state they assume poisoned samples remain separable in representations fitted to poisoned labels, and adaptive adversaries can break that space (Qi et al., ICLR 2023; Tan and Shokri, 2020). Newer members read activation gradients or use sharpness-aware minimization to stabilize detection. The authors themselves ran spectral signature and activation clustering, keeping the statistic and changing the space.
• Other pipeline positions: pruning, Neural Cleanse, universal litmus test, STRIP, SentiNet, Scale-Up act on the model or at inference. The authors state that explanation-based forensics require a trained model and triggers in images, and labels-only attacks provide neither. There is also training-time disentanglement, distilling away triggers, and work giving guarantees rather than heuristics; the authors mention that stronger attacks can break cleaning without those guarantees.

Label error detection and frozen encoders

• Confident Learning: Northcutt et al. (2021) is the standard label error method and also the authors' recent published baseline. The authors state that on the same features, it recovers fewer poisoned samples than its AUROC suggests. Loss and margin filtering find bad labels from the model's own fit, and the authors evaluate one of them in Section 5. Crowdsourcing literature treats noisy labels as an estimation problem, the setting assumed in FLIP's first scenario.
• Frozen public encoders: the authors use DINOv2 (Oquab et al., 2024). They state that such auditing is becoming common, and concurrent work uses similar reasoning to find backdoors in CLIP data. The authors claim their addition is not the encoder but fixed auditing, varying only the space, with clean-label models as control.
• Distillation: FLIP's second scenario. Distillation has been proposed as a defense, but backdoors can survive distillation. The authors state they did not evaluate that scenario.

Baselines and benchmarks: SPECTRE, spectral signature, activation clustering, Deep k-NN, Confident Learning, loss filtering; benchmark is CIFAR-10, labels from FLIP's released set and local same pipeline. The authors position this paper as: not proposing a new detection algorithm, nor claiming resistance to adaptive attackers; the goal is to separate space, statistic, and deletion rule, and select audits using retrained models rather than detection scores.

Do not create new detectors. Figure 1 splits auditing into three knobs: feature space, statistic, and deletion rule. Turn one at a time and retrain the victim, evaluate with CTA and PTA, not by ranking scores.

How the attack only changes labels

• Three steps: train an expert on poisoned copies with triggers and flipped labels, record trajectories; use trajectory matching to optimize soft labels on clean images so the student follows that trajectory; then round soft labels into hard flips.
• Which labels to select: the score for each sample is the maximum logit of the wrong class minus the logit of the correct class. Keep the highest m, flip them to that sample's own wrong-class argmax.
• Two points the authors emphasize: scores are computed over the entire training set, not pushing flips into the source class; the new label is the expert's preferred wrong class at the time, not a fixed target. At m=1500, truck→deer has 1297 flips on truck images and 1209 labeled as deer; dog→cat is 1416 and 1499. Assuming poisoned samples are all in one class or share a single label is something the attack does not promise.

Two types of statistics

• Input: N×D L2-normalized features and dataset-provided labels. No access to true labels, clean subsets, or poison masks.
• Nearest neighbor: k-nearest neighbors by cosine similarity, excluding self. d(i)=1/k∑_{j∈ Nk(i)}1[ỹ_j≠ỹ_i], i.e., knn_disagreement, the proportion of neighbors with different labels. knn_sim_margin is a continuous variant preserving locality. The k-NN numbers in the text use k=20.
• Prototype: p(i)=max_c⟨ f_i,μ_c⟩−⟨ f_i,μ_ỹi⟩, i.e., how much better the other class centroids are than the claimed class. Centroids use dataset-provided labels, and the target class centroid is polluted by flipped-in samples. prototype_trimmed drops the 20% farthest from the temporary centroid and recomputes; the proportion is fixed at 0.2, not scanned by the authors.
• Ties: at k=20 there are only 21 distinct scores. At the maximum 1.0, truck→deer has 396 samples and dog→cat has 156, ties broken by dataset order, and the budget is spent on this end. Prototype scores tie at the other end: 91.9% on truck→deer and 92.3% on dog→cat are exactly 0, unreachable by the budget; the actually read ranking spans about 4000 values.

Four feature spaces

• Frozen: DINOv2 ViT-S/14 (384-dim, 22.1M) and ViT-B/14 (768-dim, 86.6M). The encoder is chosen before seeing this data, frozen, and the training labels of this dataset cannot reach it.
• Supervised: ResNet-32 penultimate layer 64-dim trained on poisoned labels, and 64-dim on the same architecture, same recipe, clean labels. Each image passes through the encoder once, deterministic evaluation transform, no augmentation, cosine similarity equals inner product.
• Control is not a defense: with clean labels already available, auditing is unnecessary. This column only separates "supervised features are unsuitable" from "features fitted to poisoned labels are unsuitable."

From scores to deletion

• Three rules: auto deletes samples above τ. Equation (4) at τ=0.5 means a majority of neighbors disagree with the given label; the authors call this a deployable operating point, and the headline mitigation numbers use it. topn deletes a fixed rank, used to compare who is deleted under matched budget. perclass deletes the highest Q proportion within each claimed class.
• Why SPECTRE must be per-class: class-calibrated scores cannot be cut globally. When the authors first gave SPECTRE the global top 2728, on dog→cat 2372 deletions fell outside the class holding 1499/1500 poisoned samples, recall 0.233; the same budget spent within that class gives 0.805. Therefore SPECTRE is always reported per-class.
• Diagnostic cut: precision@k cut by true poison count is only for comparison with prior work, not an operating point. Appendix B states that prototype_trimmed's budget in Table III is converted from equation (4), not a prototype-score cut at τ=0.5.

Pre-training covert flip pool

• Definition: for ordered pair a→b, hypothetically relabel each training image of class a to b and look at the equation (4) score. That score is 1 minus the proportion of neighbors already labeled b. The pool is the number of class-a images whose hypothetical score does not exceed τ. Reported at τ=0.5.
• Input: only encoder and clean labels. No attack, poisoned labels, or training. The authors computed this pool for all 90 ordered pairs before running any attack in this paper.
• Authors' positioning: the defender does not have clean labels, so the pool is not a screening tool run on suspicious datasets but an analysis tool. Relative to clean neighbor labels, it is a lower bound because flips in the same region cover each other.

Experimental Setup

• Data and victims: CIFAR-10, 50000 training images, 10000 test images, 10 classes. ResNet-32 trained from scratch for 200 epochs, SGD, learning rate 0.1, Nesterov momentum 0.9, weight decay 2×10−4, batch size 256, learning rate divided by 10 at epochs 75 and 125. Defended and undefended use the same recipe.
• Attack: FLIP, m=1500. Trigger is a sinusoidal pattern and Turner corner block; the authors state that every claim in the main text uses the sinusoidal. truck→deer uses the label released by FLIP; dog→cat, car→truck, cat→dog are generated locally by the same pipeline.
• Encoder and baselines: Auditing uses frozen DINOv2 ViT-S/14, with ViT-B/14 for scale ablation. Baselines are SPECTRE, activation clustering, spectral signature, Confident Learning, loss/margin filtering, and nearest-neighbor and prototype variants.
• Metrics: CTA is accuracy on the unmodified test set. PTA per equation (3) is a proportion, with denominator the 1000 images of the source class. The introduction writes SPECTRE as 1.7% attack success rate and directly contrasts it with 0.97 in Table III. The clean-label floor is written in the main text as truck→deer 0.001, dog→cat 0.061. Detection uses AUROC, AUPRC.
• Replication: Load numbers read from trained models are mean ± standard deviation over three seeds. Detection metrics are deterministic given the feature cache and label set. The Turner row in Table I is a single run.
• Reproduction: Table I clean-label CTA 92.57, PTA 0.1, published 92.38, 0.1. Sinusoidal behavior 89.47±0.35, 99.73±0.15, published 89.87, 99.8. Turner 89.94, 32.4, published 89.91, 99.9. The authors state this is a single-corner trigger, which does not match the published four-corner result, and they do not draw mitigation conclusions from it.

Main Results

Per Table III, ResNet-32, sinusoidal FLIP, m=1500, three training seeds. SPECTRE uses its per-class budget as designed.

Swipe horizontally to read the table

Defense and class pair	Deleted images	CTA	PTA
none, truck→deer	0	89.47±0.35	99.73±0.15
SPECTRE, Q=0.20, truck→deer	10000	90.94±0.23	0.97±0.46
prototype_trimmed, truck→deer	2981	92.23±0.14	0.20±0.20
none, dog→cat	0	89.33±0.30	94.67±0.42
SPECTRE, Q=0.20, dog→cat	10000	89.12±0.29	51.30±0.89
prototype_trimmed, dog→cat	2728	91.88±0.18	24.30±2.51

• Authors' interpretation: On truck→deer the attack is removed, and clean accuracy returns to within half a point of the clean-label model's 92.57; on dog→cat it is reduced but not removed, which the authors describe as partial defense.
• Compared with published SPECTRE: The authors state they reproduce 90.94/0.97, against FLIP Table 16's 91.47/1.7, so this is pricing SPECTRE. They state it deletes 20% of each class, the audit cut is 3.4× smaller, and it retains 1.3 more clean accuracy points.
• Harder pair: The authors state that on dog→cat, under the same 10000-image budget, the gap on both axes is larger. SPECTRE retraining results for car→truck and cat→dog are not reported.

Only Changing the Feature Space

• AUPRC: Table II is written as truck→deer/dog→cat. prototype_trimmed in the poisoned supervised space is .27/.04, clean-label baseline is 1.0/1.0, frozen ViT-S/14 is .98/.73, ViT-B/14 is .99/.87. knn_disagreement is .26/.05, .97/.86, .61/.35, .74/.52. The authors state the between-group gap is larger than the within-group gap.
• AUROC: The main text and Figure 1 write that on dog→cat, knn_disagreement is 0.546 (random 0.5), 0.967, 0.995, respectively poisoned features, frozen DINOv2, clean-label baseline. Section 5 writes prototype audit on poisoned features is 0.511. The authors state the deficit is in fitting poisoned labels, not in the supervised features themselves; the clean column is an upper bound, not a method. The main text also writes clean-column AUPRC 0.966 and 0.864, and Table II's corresponding cells are printed as .97 and .86.
• Testing and scale: Under prototype_trimmed, ViT-S/14's AUPRC gain over the poisoned space is +0.710 and +0.686, 95% intervals [+0.688, +0.730], [+0.661, +0.708], B=1000, the authors report p<10−3. ViT-B/14 further improves by +0.018 and +0.143, intervals [+0.013, +0.022], [+0.126, +0.159], without changing the ranking.

Detection Scores, Retraining, and Adaptive Attacker

• Matched budget: Table V does not mark seeds. On dog→cat, knn_sim_margin's AUROC/AUPRC is .971/.424, PTA 56.9; knn_disagreement is .967/.348, PTA 53.6. prototype_trimmed's AUPRC .727 is higher than prototype_margin's .673, but PTA is 25.7 vs 23.0. The main text also writes on truck→deer 11.4 vs 8.6. The authors state that relying only on detection scores had previously led them to treat a continuous variant as an improvement, but after retraining it was not.
• Confident Learning: In Table V, on dog→cat, cleanlab AUROC .959 vs prototype .951, AUPRC .353 vs .727, PTA 67.3 vs 25.7. The authors state this is a 2.6× attack success rate. On truck→deer, cleanlab's PTA is 51.3, prototype's is 0.2. car→truck's AUROC is 0.961 vs 0.959, AUPRC 0.480 vs 0.789; cat→dog's AUROC differs by 0.001 (0.969 vs 0.970), AUPRC by 0.29 (0.494 vs 0.786). The authors state that on two of the four pairs, using the commonly used AUROC would select a detector that leaves more than twice the attack success rate.
• After knowing the audit: When targeting the prototype audit, it recovers 1.4% of placed flips on dog→cat, while the nearest-neighbor audit recovers 58.5%. Under the same stealth budget, 5 flips remain under the nearest-neighbor score, 579 under the prototype score. When targeting the nearest neighbor and switching to prototype, PTA is 15.77±0.15 vs 13.77±0.81, p=0.046. When targeting the prototype and switching to nearest neighbor, it is 22.20±1.74 vs 24.17±0.40, p=0.185, three seeds are indistinguishable. Figure 6's caption also writes 0.234 vs 0.237, which differs from this set of numbers in the main text. In the column where the prototype is targeted, the off-diagonal defense catches 45.0% of placed flips, the matched defense catches 28.4%; the authors state the models differ by only about two PTA points. Figure 5 notes that under the stealth constraint only 674/1377 flips are placed (budget 1500).

Pool, Deletion Cost, and Time

• Pool and retraining: Table VI's pool, AUPRC, no-defense PTA, post-defense PTA are: truck→deer 0, .976, 99.7±0.2, 0.2±0.2; car→truck 72, .789, 99.4±0.3, 24.6±1.3; cat→dog 82, .786, 94.4±0.7, 24.4±2.7; dog→cat 265, .727, 94.7±0.4, 24.3±2.5. The authors state this is a step, not a slope; the zero side has only one pair measured. Pool monotonically corresponds to AUPRC, but cannot rank post-defense PTA for the three nonzero pairs. In Figure 7, truck→deer goes from 99.9 unconstrained to 1.7 at τ=0.9, 0.2 at τ=0.5. The authors state the collapse is because there are no steps to take, not because the defense is stronger on that pair.
• Catching 90% of poisoned samples: Table IV, on truck→deer, SPECTRE deletes 10000, destroying 8555 clean images; activation clustering 8630; spectral signature Q=0.25 deletes 12500, destroying 11145; prototype_trimmed deletes 1419, destroying 69. On dog→cat, SPECTRE at Q=0.50 is 0.849, spectral signature 0.764, neither reaches 90%; activation clustering Q=0.50 deletes 25001, destroying 23617; prototype deletes 3432, destroying 2082. If the target class is given to SPECTRE, dog→cat recall is 0.828, destroying 1739 clean images, vs the audit's 0.900 and 2082. The authors state the advantage is in real deployment and the easier pair, and do not claim more.
• Time: Table VII, ViT-S/14 forward pass 277.6 seconds, 8.2% of the training median 3372 seconds (n=99); ViT-B/14 is 1190.0 seconds, 35.3%. The authors write the full pipeline as frozen-encoder audit 3650 seconds, 1.08 training runs, SPECTRE route 6746 seconds, 2.00 training runs, and state the ratio should be read.

Other Ablations and Analyses

• Loss filtering: cross-entropy AUROC/AUPRC on truck→deer is 0.893/0.355, on dog→cat 0.757/0.102; logit margin is 0.891/0.333 and 0.766/0.107. Catching 90% of poisoned samples deletes 14382 and 29393 clean images, vs prototype audit's 69 and 2082.
• k scan: Figure 4 does 20 victim trainings on truck→deer. AUPRC peaks at k=200; k=500's CTA/PTA is 92.36/0.3, k=200's is 92.27/0.7. Auto threshold deletes 3827 vs 4425 images.
• Statistics that do not read labels: activation clustering within the target class on dog→cat, poisoned features AUROC 0.903, on DINOv2 0.122; 2-means split is 47.7/52.3. The authors state the claim covers statistics that compare samples with their claimed labels.
• Deployable thresholds: Appendix B, knn_disagreement at τ=0.5 recall on truck→deer is 91.3% (marking 2981), on dog→cat 67.7% (marking 2728). Otsu on dog→cat recall 95.7%, marking 5917 images; the authors did not train a victim at that cutoff.
• Direction: pools are 265 and 82, post-defense PTA is 24.3±2.5 and 24.4±2.7 (main text and Table VI). The authors state direction affects geometry, not the victim.
• Another timing: on the same cluster, k-NN graph 31.2 seconds, SPECTRE 1.56 seconds, vs Table VII's 36.4 and 2.2.

Limitations and Future Directions Noted by the Authors

• Scope: one dataset, one architecture, one effective trigger, one budget, namely CIFAR-10, ResNet-32, sinusoidal trigger, m=1500. The authors claim no evidence for more classes, larger images, other architectures, or budgets far from 3%; code supports FLIP's other datasets and architectures, but only one each was run, the gap being compute (Section X).
• Pairs not reproduced or not run: Turner not reproduced. The authors treat it as a partial result and draw no mitigation conclusions from it. By pool ordering, the hardest four pairs do not include truck→car, which was not run (Section X).
• Encoder has seen such images: DINOv2 zero-shot k-NN accuracy on CIFAR-10 is 96.3%. The authors claim the audit is label-independent but not knowledge-independent; on medical scans, satellite imagery, and industrial inspection, the neighborhoods carrying signal may not form, and this is untested (Section X).
• Choices not swept: pruning ratio 0.2 is the first value tried, the authors claim it should be swept. k-NN numbers on harder pairs carry a tuning advantage a real defender would not have; prototype audit has no k. The 90% recall comparison uses a true mask, not an operating point; on dog→cat the deployable threshold recall is 67.7%, so the deployed audit does not reach 90% (Section X).
• Only deletion tested: no label correction, downweighting, robust losses, or soft labels from the attack output. The authors claim deletion saturates, so this is the next thing to try. The adaptive attacker does not re-optimize the expert; every adaptive number is an upper bound for the defense. The pool only separates "no stealthy attack" from "has one," not ranking the latter; of 90 pairs, 29 have zero pool, only one was trained, and relative to clean neighbor labels it is a lower bound (Section X).
• Next steps listed by the authors: sweep pruning ratio; test on CIFAR-100 and Tiny-ImageNet with ResNet-18 and VGG-19 whether "space matters" still holds; use pretrained out-of-distribution data to separate label independence from encoder familiarity; let the attacker re-optimize the expert under the stealth constraint; test on more than four pairs whether the pool predicts retraining results (Section XI).

Experimental Coverage

• Victim is ResNet-32, data CIFAR-10, sinusoidal trigger, m=1500; class pairs truck→deer, dog→cat, car→truck, cat→dog. Stealthy-flip pool computed for all 90 ordered pairs, victim training covers 4 pairs, 1 of 29 zero-pool pairs trained (Sections IV, VII, X).
• Baselines include SPECTRE, activation clustering, spectral signatures, Confident Learning, loss filtering, and Deep k-NN-style nearest-neighbor criteria; SPECTRE uses per-class budget as designed (Sections III, V, VI).
• Numbers read from trained models are over three training seeds; detection metrics are deterministic given the feature cache and label set. Turner is a single run (Section IV, Table I).
• Adaptive experiments on dog→cat, both sides use the same scoring code, each deletes 5%, three seeds (Section VI, Figure 6).
• The paper does not report victim retraining at the Otsu cutoff, does not report pruning ratio sweep, does not report victim results for truck→car (Appendix B, Section X).

The authors ask: can a defender without trusted data decide which training labels to keep, and how much clean data does this destroy, and what is the final model accuracy?

• Question: FLIP flips only 3% of labels on CIFAR-10, without modifying images. The authors claim ResNet-32 classifies 99.7% of triggered source-class test images into the attacker class. Common defenses, however, look for poisoned samples in representations that have already fit these labels.
• Approach: Fix the detector, labels, and metric, and only change the feature space. Compute nearest-neighbor disagreement or class prototype scores on frozen DINOv2, delete, then retrain ResNet-32. Features from the same architecture on clean labels serve only as a control. Evaluation uses retrained CTA and PTA, not AUROC as a substitute.
• Space: The authors report AUROC for knn_disagreement on dog→cat: poisoned features 0.546, frozen features 0.967, clean-label control 0.995. The authors claim the deficit comes from fitting poisoned labels, not from the supervised features themselves.
• Retraining: In Table III, on truck→deer, prototype_trimmed deletes 2981 images and PTA is 0.20±0.20; SPECTRE deletes 10000 and PTA is 0.97±0.46. The authors claim this contrasts with a reproduced SPECTRE (90.94/0.97 vs. FLIP-reported 91.47/1.7). On dog→cat, this audit's PTA is 24.30±2.51. In Table VI, the other two non-zero-pool class pairs have post-defense PTA of 24.6±1.3 and 24.4±2.7.
• Authors' conclusion: Detection quality is not a proxy for protection. The authors claim that on two of four pairs, Confident Learning has higher AUROC but leaves 2.6 times the attack success rate on dog→cat; the prototype family, which wins every static comparison, is the first to be bypassed by an attacker who knows the audit. Past a certain point, catching more poisoned samples no longer reduces attack success rate. The conclusion states: inspect labels in a space that has never learned these labels, and judge defenses with retrained models rather than detection scores.