Skip to content

A general framework for detecting anomalous inputs to DNN classifiers

A General Framework For Detecting Anomalous Inputs to DNN Classifiers

arXiv· arXiv:2007.15147· Jayaram Raghuram, Varun Chandrasekaran, Somesh Jha, Suman BanerjeeComputer Sciences, University of Wisconsin, Madison; XaiPient Inc.Indexed by this site · 23 days ago· Originally publishedDefenses

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

The authors propose the unsupervised anomalous-input detection framework JTLA and compare it with five methods on CIFAR-10, SVHN, MNIST; the authors claim it outperforms the baselines in most attack and OOD settings.

Threat modelThe defender holds a trained DNN classifier and a labeled dataset D containing no anomalous samples. The attacker is gray-box: fully aware of the DNN architecture and parameters, partially aware of the detection method (e.g. the detection threshold and the layers used may be unknown); the attack targets test-time evasion, including PGD, CW, FGSM and the defense-aware adaptive attack proposed by the authors.

Questions
DNN classifiers err with high confidence on adversarial examples and OOD inputs; the authors argue that existing layer-representation-based detection methods lack a unified, principled framework, are mostly heuristic, and often require known anomalous samples for training and struggle to generalize to unknown or adaptive attacks.
Methods
The authors propose the unsupervised meta-algorithm JTLA: at each layer compute test statistics conditioned on the predicted class and on each candidate true class (a multinomial LRT based on kNN class counts), normalize with p-values, then aggregate and score via Fisher or aK-LPE, with decision by FPR threshold; they also propose a gray-box adaptive attack against kNN class-count defenses.
Experiments and results
Compared with five methods on CIFAR-10, SVHN, MNIST, the authors claim JTLA is better in most adversarial and OOD settings; exceptions include higher average precision for Odds under the SVHN adaptive attack, OOD detection inferior to the supervised Mahalanobis, and Odds and Trust comparable or slightly better under some MNIST attacks.
Limitations and future work
The authors note that aK-LPE multivariate p-value estimation requires large samples, and class-conditional sample sizes (500–5000) may introduce estimation error; the Fisher method assumes p-values are independent. Experiments cover only three image datasets, five baselines, and four attacks, and the paper does not have a dedicated Limitations section discussing limitations.
Experimental setupResnet (CIFAR-10)、CNN (SVHN) et al. · CIFAR-10、SVHN et al. · average precision、pAUC-α (pAUC-0.2, pAUC-0.05)
Evaluated models
Resnet (CIFAR-10)CNN (SVHN)CNN (MNIST)
Benchmarks
CIFAR-10SVHNMNISTNot-MNISTCIFAR-100
Metrics
average precisionpAUC-α (pAUC-0.2, pAUC-0.05)
AI summaryThe researchers propose an unsupervised anomaly detection framework based on the internal layer representations of DNNs, using a meta-algorithm of configurable components to uniformly detect adversarial examples and out-of-distribution (OOD) inputs. The framework provides concrete instantiations of each component of the meta-algorithm based on statistical tests and anomaly detection ideas. The authors evaluate on well-known image classification datasets using strong adversarial attacks and OOD inputs, including an adaptive attack that exploits the internal layer representations of DNNs, a type of attack often not considered in prior work. The authors state that comparison with five recently proposed detection methods shows the effectiveness of the method in detecting adversarial and OOD inputs.

The researchers propose an unsupervised anomaly detection framework based on the internal layer representations of DNNs, using a meta-algorithm of configurable components to uniformly detect adversarial examples and out-of-distribution (OOD) inputs. The framework provides concrete instantiations of each component of the meta-algorithm based on statistical tests and anomaly detection ideas. The authors evaluate on well-known image classification datasets using strong adversarial attacks and OOD inputs, including an adaptive attack that exploits the internal layer representations of DNNs, a type of attack often not considered in prior work. The authors state that comparison with five recently proposed detection methods shows the effectiveness of the method in detecting adversarial and OOD inputs.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    To detect adversarial and OOD inputs for DNN classifiers, the authors propose a unified unsupervised layer-representation detection framework, JTLA.

    Core problem: how to detect adversarial examples and OOD inputs in a unified, principled and unsupervised way based on DNN internal layer representations.

    • Problem setting and importance: DNNs may err with high confidence on inputs outside the training distribution; anomalous inputs may come from external factors or malicious attackers, and the authors argue that deployed classifiers need detection mechanisms to reject or give more reliable predictions.
    • Shortcomings of existing methods (per the paper): supervised methods require known anomalous samples for training and often fail to generalize to unknown or adaptive attacks; methods using only inputs, outputs or a single layer do not jointly exploit information across layers; multi-layer methods lack a pluggable unified framework; unsupervised methods based on density modeling struggle with high-dimensional layer representations; many methods do not exploit the class-conditional nature of the predicted class.
    • Core observation: adversarial examples are anomalous relative to the natural input distribution of the predicted class, but typical relative to the distribution of their source class; JTLA uses this assumption to design the adversarial detection scoring function (Appendix A.3 notes that Miller et al., 2019 used a similar assumption).
    • What the paper proposes: the unsupervised meta-algorithm JTLA (four configurable components: test statistic, normalization, aggregation, scoring) and a concrete implementation based on statistical testing; a defense-aware adaptive attack against kNN class-count defenses; comparison experiments with five methods on three image datasets.
    • Threat model:
      • Attacker goal: test-time evasion—add perturbation to clean inputs to make the DNN misclassify while evading kNN class-count-based detection.
      • Attacker knowledge: gray-box—fully aware of the DNN architecture and parameters, partially aware of the detection method (the paper gives examples: the detection threshold and the layers used may be unknown).
      • Attacker capability: generate PGD, CW, FGSM white-box adversarial examples, and the paper's adaptive attack that optimizes layer-representation kNN class counts.
      • Victim system and defender assumptions: a trained DNN classifier; the defender additionally has a labeled dataset D, different from the training set and containing no anomalous samples, for estimating detector parameters.
  2. What related research exists?

    Related work is divided into three groups: supervised detection, unsupervised detection and confidence measures, plus adaptive attack research and five comparison baselines.

    Supervised detection methods

    • Mahalanobis (Lee et al., 2018): models layer representations class-conditionally with multivariate Gaussians and trains a logistic regression classifier on the Mahalanobis distance as a feature. The paper notes that its parametric model may not suit intermediate-layer representations and that it does not use the DNN's predicted class.
    • LID (Ma et al., 2018): trains logistic regression with the local intrinsic dimensionality of layer representations as the statistic. The paper notes that its statistic is not class-conditional and is not normalized.
    • Yang et al., 2020 (ML-LOO): trains a classifier with the dispersion of feature attributions (IQR, MAD) as the statistic; the paper likewise notes that its statistic is not class-conditional and is not normalized.

    Unsupervised detection methods

    • Odds (Roth et al., 2019): constructs a statistic based on the log-odds ratio of noise-perturbed inputs, with z-score normalization followed by threshold decision; the paper notes that it does not use multi-layer representations.
    • Zheng & Hong, 2018 and Miller et al., 2019: model class-conditional densities of layer representations with Gaussian mixtures; the paper notes that high-dimensional layer representations are hard to model accurately with such density models.
    • Sastry & Oore, 2020: detects OOD with a Gram-matrix-based deviation statistic; the paper notes that this statistic can serve as a pluggable test statistic for JTLA.

    Confidence measures

    • Trust Score (Jiang et al., 2018) and Deep kNN (Papernot & McDaniel, 2018): provide an independent confidence for classifier predictions, and low-confidence inputs are flagged; the paper notes that DKNN's statistic based on kNN class counts of the predicted class can be seen as a binomial special case of JTLA's multinomial statistic, and both can be classified as unsupervised methods.

    Adaptive attack research

    • Carlini & Wagner, 2017a and Tramèr et al., 2020: the paper cites their conclusion that most existing detection methods cannot cope with unseen and adaptive attackers who know the defense mechanism, to justify evaluating defense-aware attacks.

    Baselines and benchmarks: the comparison baselines are five—Mahalanobis, LID, Odds, DKNN, Trust (LID and Odds do not participate in the OOD experiments); the datasets are CIFAR-10, SVHN, MNIST, and the OOD experiments use MNIST vs. Not-MNIST, CIFAR-10 vs. SVHN, CIFAR-10 vs. CIFAR-100.

    The authors position this paper as: bringing the above layer-representation-based methods into a unified meta-algorithm with configurable test statistic, normalization, aggregation and scoring, and filling in each component with class-conditional p-value normalization and multiple-testing aggregation.

  3. How does the paper address this problem?

    JTLA meta-algorithm: class-conditional test statistics → p-value normalization → cross-layer aggregation → scoring and decision by FPR threshold.

    The overall idea is to decompose anomaly detection into a meta-algorithm JTLA (Joint statistical Testing across DNN Layers for Anomalies) with four configurable components, and to give each component an implementation based on statistical testing; the detector estimates parameters only from a natural-sample anomaly-free labeled set D and does not need known anomalous samples.

    Problem Setting

    • The defender holds a trained m-class DNN classifier F and a labeled dataset D (different from the training set, containing no anomalous samples); passing samples in D through the network yields layer representations x(`) and predicted classes, forming an augmented dataset Da, partitioned into subsets by layer × predicted class and layer × true class.
    • For a test input with predicted class ĉ, m+1 class-conditional test statistics are computed at each layer: one conditioned on the predicted class, and m conditioned on each candidate true class.

    Test Statistic: Polynomial LRT of kNN Class Counts

    • Take the k nearest neighbors of the layer ` representation and count the classes (k1,…,km); the class counts of natural inputs are modeled with a multinomial distribution, with parameters estimated by MAP with a Dirichlet prior from the corresponding data subset.
    • A multinomial likelihood ratio test measures the deviation: T(x(ℓ), ĉ, D̂a(ℓ, ĉ)) = ∑i=1m ki(ℓ) log (ki(ℓ))/(k πi(ℓ)), where a larger value means a larger deviation from the natural distribution; the candidate-true-class version is analogous.

    Normalization and Aggregation

    • Normalization: class-conditional p-values turn each layer's statistic into a probability approximately uniform for natural inputs; scheme A estimates right-tail p-values for single-layer and layer-pair statistics using the empirical CDF (averaged over 100 bootstraps to reduce variance), scheme B estimates aK-LPE multivariate p-values for the whole vector of layer statistics (Qian & Saligrama, 2012).
    • Aggregation: scheme A combines the layer and layer-pair p-values with Fisher's method $ \log q_{fis}(t) = \sum_{q \in Q} \log q $ (an HMP method suitable for correlated p-values is also given, Appendix B); scheme B requires no aggregation.

    Scoring and Decision

    • Adversarial score: S = log (maxc ≠ ĉ qagg(ts|c))/(qagg(tp|ĉ)), where the numerator takes the largest candidate-true-class aggregated p-value among non-predicted classes; the OOD score simplifies to −log qagg(tp|ĉ).
    • Decision: a score above the threshold is judged anomalous; the threshold is determined by the empirical FPR function estimated on Da according to the target FPR α (the paper gives 0.01 as an example).
    • Implementation: NPP is used to reduce the dimensionality of layer representations (the logit layer is not reduced), and NNDescent is used for approximate nearest neighbors; the only hyperparameter k is set by the heuristic k = n^0.4 (n is the number of in-distribution training samples).

    Adaptive Attack (§5)

    • Against defenses relying on kNN class counts: the attacker optimizes the perturbation to minimize the cross-layer cumulative class-count log ratio between the source class and the target class, softening the counts with a Gaussian kernel and optimizing with RMSProp (at most 1000 steps), with λ chosen by a ten-step binary search as the smallest value that produces a successful adversarial example; the target class is the class with the second-highest DNN predicted probability.
  4. What experiments did the paper conduct?

    Three datasets, five baselines, four attacks and three sets of OOD tasks; the authors state that JTLA is best-performing in most settings, with several exceptions.

    Experimental Setup

    • Models under test: ResNet-34 for CIFAR-10 (test accuracy 95.45%), 2 Conv + 3 FC for SVHN (89.42%), 2 Conv + 2 FC for MNIST (99.12%) (Table 2).
    • Datasets: CIFAR-10, SVHN, MNIST; OOD tasks are MNIST vs. Not-MNIST, CIFAR-10 vs. SVHN, CIFAR-10 vs. CIFAR-100 (plus random pixel samples).
    • Baselines: Mahalanobis, LID (supervised, using adversarial/anomalous data in the training fold), Odds, DKNN, Trust (unsupervised); LID and Odds do not participate in the OOD experiments.
    • Attacks: PGD (ε=1/255), CW (confidence=0), FGSM (εmax=1) and the paper's adaptive attack (targeting the JTLA Fisher variant), generated with Foolbox.
    • Metrics and protocol: average precision and pAUC-α (pAUC-0.2 for adversarial, pAUC-0.05 for OOD), no threshold needed; 5-fold class-stratified cross-validation on the test set, detector parameters estimated on the training fold and metrics computed on the test fold; when reported as a proportion, the median of 100 random subsets is taken.
    • JTLA variants: two kinds, Fisher aggregation of layer + layer-pair p-value normalization, and aK-LPE multivariate p-values, both using polynomial statistics.

    Main Results

    Main-text results are presented as curves (Figure 2, 3, 4), and the converted text has no numerical values; the table below takes the average precision / pAUC-0.2 of each JTLA configuration from the appendix ablation table Table 6 (including all adversarial examples, SVHN and CIFAR-10):

    Swipe horizontally to read the table

    ConfigurationSVHN, PGDSVHN, CWCIFAR-10, PGDCIFAR-10, CW
    Layer + layer-pair p-values + Fisher0.7382 / 0.80250.9631 / 0.92130.7710 / 0.77900.9664 / 0.9377
    Layer + layer-pair p-values + HMP0.7296 / 0.79660.9611 / 0.91710.7614 / 0.77050.9653 / 0.9344
    Layer p-values only + Fisher0.7393 / 0.80050.9634 / 0.92140.7734 / 0.77810.9667 / 0.9380
    Multivariate p-values (aK-LPE)0.7161 / 0.78400.9559 / 0.90420.7437 / 0.75180.9650 / 0.9296

    (According to Table 6, the metrics are average precision / pAUC-0.2; PGD uses ε=1/255, CW uses confidence=0)

    • The authors claim that JTLA outperforms the other baselines in almost all cases in Figure 2, 3; Mahalanobis, Odds, DKNN do well in some cases and fail in others, and LID performs poorly in almost all cases.
    • Exception: in Figure 3b under the SVHN adaptive attack, Odds has higher average precision than JTLA; the authors claim that under the same setting pAUC-0.2 (Figure 10) is higher for JTLA in the low perturbation norm range.
    • The authors believe Fisher outperforms aK-LPE because multivariate p-value estimation requires large samples, while the class-conditional sample size is only 500–5000.

    OOD Detection

    • What was measured: three dataset pairs, comparing JTLA with Mahalanobis, DKNN, Trust on average precision and pAUC-0.05 (Figure 4).
    • Results: the authors claim JTLA outperforms the unsupervised DKNN and Trust in all cases, but is inferior to Mahalanobis; Mahalanobis used outlier data in the training fold, whereas in real scenarios such prior samples are usually unavailable.

    Attack Transfer and FGSM, MNIST

    • CW→PGD transfer (Figure 6): the authors claim Mahalanobis and the two JTLA variants are best on CIFAR-10, the two JTLA variants outperform other methods on SVHN, and on MNIST the aK-LPE variant and Trust are best; LID is poor on all datasets.
    • PGD→CW transfer (Figure 7): the authors claim that on CIFAR-10 Odds is best, followed by the two JTLA variants; on SVHN JTLA is best; on MNIST JTLA, Odds, Trust are close. The authors speculate that Odds adapts to the CW attack because its statistic is based on the class-pair log-odds ratio.
    • FGSM (Figure 8) and MNIST (Figure 11, 12): the authors claim that under FGSM the two JTLA variants outperform other methods; on MNIST under CW and FGSM attacks Odds, Trust and JTLA are comparable or slightly better, and under the adaptive attack the Fisher variant's performance degrades as the perturbation decreases while the aK-LPE variant outperforms other methods, which the authors attribute to the adaptive attack being optimized against the Fisher variant.

    Other Ablations and Analysis

    • Number of layers (Table 7): generally the more layers the better the performance; on CIFAR-10 under CW attack using only the logit layer is comparable to using all layers (0.9664/0.9371 vs 0.9664/0.9377).
    • Scoring function (Table 8): the adversarial scoring function outperforms the OOD scoring function under all normalization × aggregation combinations (e.g. Fisher: SVHN PGD 0.7382 vs 0.7078).
    • Layer pairs (Table 6): adding layer pairs brings no clear improvement (e.g. SVHN PGD Fisher: 0.7382 vs layer-only 0.7393).
    • Runtime (Table 9): Trust is fastest (0.53 minutes per fold on CIFAR-10), JTLA Fisher 2.73 minutes, Odds 142.94 minutes and LID 49.74 minutes are the slowest.
  5. What are some points that can be further explored?

    The paper has no dedicated limitations section; in the ablation discussion the authors point out issues such as the sample size requirement of aK-LPE and the independence assumption of Fisher.

    Limitations and future directions noted by the authors

    • aK-LPE estimated sample size: Appendix E.2 notes that multivariate p-values require a large sample to converge to the true value, and that the class-conditional sample size (500–5000 in the experiments) may cause estimation error.
    • Independence assumption of the Fisher method: §4.3 notes that the Fisher method's assumption that the p-values of the individual tests are independent is its "obvious weakness," and gives HMP, applicable to correlated p-values, as an alternative.
    • Layer-pair overhead: §3.2 explains that once the number of layers exceeds a few dozen, including all layer pairs is inefficient, so layer pairs are optional in Algorithm 1.
    • Realism of the Mahalanobis comparison: §6.2 notes that in real scenarios it is unlikely that prior knowledge and a sufficient number and variety of outlier samples would be available for training, implying the conditions under which the supervised comparison method applies.
    • The paper has no dedicated Limitations or Future Work section; the above are limitations the authors explicitly state in the method design and experimental discussion.

    Scope of experimental coverage

    • The models evaluated are three CNNs: ResNet-34 (CIFAR-10), 2 Conv + 3 FC (SVHN), 2 Conv + 2 FC (MNIST), all 10-class image classification (Table 2).
    • Attack evaluation covers PGD, CW, FGSM and adaptive attacks, including two attack transfer combinations (CW→PGD, PGD→CW) and variations in attack strength (Appendix E.1).
    • The adaptive attack optimizes only against the Fisher + polynomial statistic variant of JTLA (Appendix D.4).
    • OOD evaluation covers MNIST vs. Not-MNIST, CIFAR-10 vs. SVHN, CIFAR-10 vs. CIFAR-100 and random pixel samples; LID and Odds did not participate in the OOD experiments.
    • The paper does not report the specific value of the detection threshold τ or the TPR at the chosen FPR operating point; the main metrics are threshold-free average precision and pAUC-α.
  6. Summarize the main content of the paper

    JTLA: an unsupervised, modular layer-representation anomaly detection framework that outperforms five baselines in most settings, accompanied by a defense-aware attack evaluation.

    A defense paper on detecting anomalous DNN inputs (ICML 2021) that proposes the unsupervised framework JTLA.

    • Problem addressed: adversarial and OOD inputs make DNNs err with high confidence; the authors argue that existing layer-representation-based detection methods lack a unified framework, rely on known anomalous samples for training, and struggle against unknown and adaptive attacks.
    • Method highlights: a four-component meta-algorithm—at each layer, compute a kNN class-count multinomial LRT statistic conditioned on the predicted class and on each candidate true class; normalize with class-conditional p-values (single-layer/layer-pair or aK-LPE multivariate); aggregate with Fisher or HMP; the adversarial score takes the log ratio of the largest aggregated p-value among non-predicted classes to the predicted-class p-value, with the threshold set by target FPR. The only hyperparameter k is set as k = n^0.4. A gray-box adaptive attack against kNN class-count defenses is also proposed.
    • Main results:
      • The authors claim that under PGD, CW, FGSM and adaptive attacks on CIFAR-10 and SVHN, JTLA has better average precision than five baselines in almost all cases (Figure 2, 3); under the ablation settings the Fisher variant's average precision is 0.7382–0.9664 (Table 6).
      • In OOD detection JTLA outperforms the unsupervised DKNN and Trust, but is inferior to the supervised Mahalanobis trained with outlier data (Figure 4).
      • Exceptions: under the SVHN adaptive attack Odds has higher average precision; under MNIST CW and FGSM, Odds and Trust are comparable or slightly better than JTLA.
      • In runtime JTLA takes 0.92–6.37 minutes per fold, lower than Odds and LID and higher than Trust (Table 9).
    • Authors' conclusion: the authors claim the modular design lets JTLA plug in various existing test statistics, and experiments under strong attacks (including adaptive attacks) support its effectiveness; the authors also note that Fisher's independence assumption and aK-LPE's sample-size requirement are limitations at the method level.

Related papers

Both use: SVHN
Show 3 more papers
Read originalarxiv.org