Skip to content

Indicators of attack failure: debugging and improving adversarial example optimization

Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples

arXiv· arXiv:2106.09947· NeurIPS· Maura Pintor, Luca Demetrio, Angelo Sotgiu, Ambra Demontis, Nicholas Carlini, Battista Biggio, Fabio RoliUniversity of Cagliari; Pluribus One; University of Genoa; Google; University of Cagliari, CINI; University of Genoa, CINIIndexed by this site · 21 days ago· Originally publishedSelected ToolsSelectedAttention38

Quick read

Analysis and theory

Based on the paper PDF (AI-generated); refer to the original

The authors propose six attack failure indicators (IoAF) to detect gradient attack optimization failures; after re-evaluating 7 defenses, most robust accuracies drop to 0–3%.

Questions
Many adversarial example defenses overestimate robustness because gradient attack optimization fails; existing evaluation guidelines are difficult to apply systematically, and automated testing and debugging tools are lacking.
Methods
The authors divide attack failures into two categories—loss landscape and attack optimization—totaling six kinds, propose six corresponding failure indicators IoAF (I1–I6) and repair strategies M1–M6, and give a semi-automated evaluation protocol.
Experiments and results
Re-evaluated 7 published defenses; after fixes, most robust accuracies dropped from 32–95% to 0–3%; all 6 models on RobustBench triggered I6; F3 errors were found in the PGD implementations of four mainstream adversarial attack libraries.
Limitations and future work
The authors state that M5 and M6 are difficult to automate, and non-triggering of indicators does not mean no stronger attack exists; I2, I6, etc. are computed only on a subset of 10 samples, and robustness evaluation uses 100 samples per model.
Experimental setupWide-ResNet (ST)、adversarially-trained model (ADV-T) et al. · MNIST、CIFAR10 et al. · Robust Accuracy (RA)、IoAF 指标 I1–I6 触发率
Evaluated models
Wide-ResNet (ST)adversarially-trained model (ADV-T)Defensive Distillation (DIST)k-Winners Take All (k-WTA)Input Transformations (IT)Ensemble Diversity (EN-DV)Turning a Weakness into a Strength (TWS)JPEG Compression (JPEG-C)Deep Neural Rejection (DNR)Stutz et al. [39]Carmon et al. [12]Sehwag et al. [37]Wu et al. [42]Ding et al. [20]Rebuffi et al. [35]MalConvDrebin (SVM-ANDROID)AUDIO-ConvNet
Benchmarks
MNISTCIFAR10Google Speech Commands Dataset(4 关键词子集)
Metrics
Robust Accuracy (RA)IoAF 指标 I1–I6 触发率
AI summaryand selection rationaleThe study proposes six indicators of attack failure (IoAF) that automatically detect why an attack fails during the optimization process of gradient attacks, together with a semi-automated pipeline for fixing them. The authors classify attack failures into loss surface and attack optimization categories, organize four known failure types, and identify two previously undiscovered failure types, while also open-sourcing the code. After validating on three task types—images, audio, and malware—and more than 15 models, they re-evaluated 7 published defenses; most original evaluations were judged to be erroneous, for example Defensive Distillation's robust accuracy dropped from 95% to 1%, and k-WTA's from 67% to 9%. Another 6 robust image models from sources such as RobustBench triggered unconstrained attack failure indicators in testing.

The study proposes six indicators of attack failure (IoAF) that automatically detect why an attack fails during the optimization process of gradient attacks, together with a semi-automated pipeline for fixing them. The authors classify attack failures into loss surface and attack optimization categories, organize four known failure types, and identify two previously undiscovered failure types, while also open-sourcing the code. After validating on three task types—images, audio, and malware—and more than 15 models, they re-evaluated 7 published defenses; most original evaluations were judged to be erroneous, for example Defensive Distillation's robust accuracy dropped from 95% to 1%, and k-WTA's from 67% to 9%. Another 6 robust image models from sources such as RobustBench triggered unconstrained attack failure indicators in testing.

Why it was selectedThe paper decomposes attack failure into automatically detectable quantitative indicators and open-sources them, re-evaluates published defenses with the same pipeline, and reveals that multiple robustness results have been overestimated.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Defense evaluation often overestimates robustness because gradient attack optimization fails; the authors propose indicators to automatically detect such failures.

    Core problem: adversarial robustness evaluation often overestimates a defense's robustness because gradient attack optimization fails, and automated testing and debugging tools to systematically find such problems are lacking.

    • Problem setting: The authors point out that many defenses fail attacks due to inappropriate attack hyperparameters (e.g., too few iterations) or gradient obfuscation, and their robustness drops substantially under stricter evaluation; even after evaluation guidelines were published, 13 defenses were still found to be mis-evaluated.
    • Shortcomings of existing methods: The authors state that existing guidelines are mostly qualitative advice and difficult to apply systematically; it is unclear whether benchmark frameworks such as AutoAttack and RobustBench are also affected by hidden attack failures.
    • Core observation: Attack failures can be mapped to specific steps of a generalized gradient attack algorithm (Algorithm 1), and thus can be classified and automatically detected by quantitative indicators.
    • Paper contributions: (i) associate four known failures with the optimization process and newly discover two failures (F3 implementation error, F6 unreachable misclassification); (ii) propose six attack failure indicators IoAF; (iii) give a semi-automated repair protocol and validate it on more than 15 models across three domains: images, audio, and malware.
  2. What related research exists?

    Related work falls into three categories: defense re-evaluation and guidelines, automated benchmarks, and the defenses and attack implementations re-evaluated in this paper.

    Robustness re-evaluation and evaluation guidelines

    • Athalye et al. [2], Tramèr et al. [41], Carlini et al. [11] — re-evaluate published defenses, propose the concept of gradient masking and qualitative evaluation guidelines; the paper notes that these guidelines are difficult to automate and are mostly ignored, and that this paper's IoAF is precisely intended to systematize these recommendations.
    • Popovic et al. [33] — the paper cites them to show that even when guidelines are followed, robustness can still be overestimated.

    Automated benchmarks

    • Ling et al. [24]'s DEEPSEC — a platform for testing multiple attacks and defenses; the paper cites Carlini [7]'s criticism that it has implementation and configuration issues.
    • Croce et al. [13, 14]'s AutoAttack and RobustBench — automatic evaluation using an ensemble of parameter-free attacks; the paper points out that their newly added warnings and black-box attack flags were inspired by this paper, but cannot guarantee finding all mis-evaluations, and that black-box attacks are computationally expensive.
    • Yao et al. [44]'s Adaptive AutoAttack — automatically searches attack configurations; the paper states that it outperforms AutoAttack but has greater computational overhead.

    Re-evaluated defenses and attack implementations (experimental subjects)

    • The 7 re-evaluated defenses: DIST [29], k-WTA [43], IT [22], EN-DV [28], TWS [45], JPEG-C [16], DNR [38], plus Wide-ResNet [47] and an adversarially trained model [21] as correctly evaluated controls.
    • Attacks and libraries: PGD [25], DeepFool [27], APGDCE/APGDDLR [13]; F3 affects the PGD implementations of four libraries: Foolbox, CleverHans, ART, Torchattacks (Table 7).

    The authors position this paper as: the authors state that this is the first testing method aimed at debugging and automatically detecting misleading robustness evaluations, complementary to benchmark frameworks — it does not run black-box attacks, and only checks the gradient attack optimization process to find such problems.

  3. How does the paper address this problem?

    Map the six types of attack failure to steps of the generalized attack algorithm, design a binary indicator IoAF and a corresponding repair for each type, then execute them in protocol order.

    Overall approach: unify gradient attacks into a generalized algorithm, attribute failures to algorithm steps, and propose Indicators of Attack Failure (IoAF), six quantitative indicators with corresponding repairs M1–M6, forming a semi-automated debugging protocol.

    Problem formalization and the generalized attack algorithm

    • The attack solves a constrained optimization: minδ L(x+δ, yt; θ), with constraints |δ|p ≤ ϵ and x+δ ∈ [0,1]d; the loss can be cross-entropy or logit loss.
    • Algorithm 1 summarizes the attack steps: initialization → optional surrogate model θ̂ → iterative gradient update → projection onto the feasible region → return the best solution along the entire path. The authors state that the method can be extended to perturbation models beyond the lp norm.

    Loss landscape failures (F1, F2)

    • F1 Shattered Gradients: components are non-differentiable or numerically unstable, making gradients impossible to compute; repair M1 uses BPDA; the new indicator I1 is set to 1 when the gradient norm is zero or computation errors occur.
    • F2 Stochastic Gradients: gradients are computable but the direction is uninformative (the model contains randomness); repair M2 uses EoT; the new indicator I2 measures the relative fluctuation V(x) of the loss in an input neighborhood, with experimental settings s=100, r=10⁻³, threshold τ=10%.

    Attack optimization failures (F3–F6)

    • F3 Implementation Errors: the authors state this is newly discovered—many attacks return the endpoint of the path rather than the optimal adversarial example, affecting PGD in four mainstream libraries and AutoAttack; M3 repairs the implementation to return the optimal point; I3 (Silent Success) is set to 1 when the endpoint is non-adversarial but an adversarial example exists along the path.
    • F4 Non-converging Attack: inappropriate step size or number of iterations; M4 increases α or n; I4 uses the cumulative minimum loss curve to compute the relative decrease D(x) over the last k iterations, with experimental settings k=10, µ=1%.
    • F5 Non-adaptive Attack: for defenses containing non-differentiable components or detectors, a surrogate model that ignores these components is used; M5 switches to an adaptive loss covering all components (the authors state this is difficult to automate); I5 is set to 1 when a sample bypasses the surrogate but not the target.
    • F6 Unreachable Misclassification: the authors state this is newly discovered—the attack gets stuck in a bad local minimum and fails even with an unconstrained attack; M6 changes the loss function (the authors state this is not easy to automate); I6 is set to 1 when the unconstrained attack fails.

    Usage protocol (Sect. 2.3)

    1. Initialize the evaluation (attack, hyperparameters, samples, model).
    2. First compute I1, I2 on a subset of N samples, and apply M1, M2 as needed (report if any sample triggers).
    3. Run the attack on all samples.
    4. Check I3, I4, I5 and rerun the affected samples after applying M3–M5; if the attack still fails, check I6 on the N-sample subset and consider M6. The authors emphasize: indicators not being triggered does not mean no stronger attack exists.
  4. What experiments did the paper conduct?

    Re-evaluated and repaired 7 defenses; most robust accuracies drop from 32–95% to 0–3%; all 6 RobustBench models trigger I6.

    Experimental setup

    • Evaluated models: 7 published defenses (DIST, k-WTA, IT, EN-DV, TWS, JPEG-C, DNR) plus 2 controls (Wide-ResNet, i.e., ST, and the adversarially trained model ADV-T); additionally 6 RobustBench models are evaluated (Stutz, Carmon, Sehwag, Wu, Ding, Rebuffi et al.).
    • Attacks: PGD/DeepFool used in the original evaluations, and AutoPGD's APGDCE and APGDDLR; repaired attacks are denoted Patched.
    • Metrics: robust accuracy RA (the proportion of samples correctly classified within ε); IoAF trigger status (I2, I3, I4, I6 with the proportion of triggered samples).
    • Scale and parameters: each evaluation uses 100 test-set samples; the subset for I1, I2, I6 is N=10; for I2, s=100, r=10⁻³, τ=10%; for I4, k=10, µ=1%; ε=8/255 for CIFAR models, ε=0.5 for MNIST models (the original DIST evaluation in Appendix A.1 states ε=0.3).
    • Implementation: SecML library; the paper does not report the number of repetitions or statistical tests.

    Main results

    Table 1: RA of each defense under the original evaluation, APGD, and patched attacks (triggered indicators omitted):

    Swipe horizontally to read the table

    ModelOriginal RAAPGDCEAPGDDLRPatched
    ST (control)0.00———
    ADV-T (control)0.48———
    DIST0.950.990.000.01
    k-WTA0.670.350.280.09
    IT0.320.120.120.00
    EN-DV0.480.000.000.00
    TWS0.770.030.680.01
    JPEG-C0.85 (Tr 0.74)0.51 (Tr)0.70 (Tr)0.01
    DNR0.750.020.200.03
    • The two control evaluations (ST, ADV-T) did not trigger any indicator, which the authors use to show that the indicators do not produce false positives on correct evaluations.
    • The authors state that after repair, most mis-evaluated defenses' RA drops to 0–3%, showing that IoAF can locate and repair errors in the original evaluation.
    • Exception: after repair, k-WTA still triggers I4, I6; the authors state that some samples can be further reduced by increasing the number of steps or the smoothing parameter; APGDDLR stops at 0.68 and 0.20 for TWS and DNR respectively and triggers I6; the authors explain that DNR's SVM-RBF kernel makes scores converge and the DLR loss saturate.

    RobustBench model re-evaluation

    • Table 2: the 6 models (Stutz, Carmon, Sehwag, Wu, Ding, Rebuffi et al.) have RA of 0.47–0.90 under APGDCE and APGDDLR, and all trigger I6 (2–10 of the 10 subset samples per model trigger).
    • The authors state that even an unbounded-perturbation optimizer cannot find an adversarial region, so the robustness claims of these models may be unreliable, and applying M6 (adaptive attack) may improve them.

    Cross-domain applicability (appendix)

    • Windows malware (A.3, Table 4): the Extend attack is reproduced on MalConv (100 samples), triggering I1, I3 (12%), I6 (3/10), RA 0.26; the authors state it can be improved with M1, M3, and that I2 requires sampling in the embedding space.
    • Android malware (A.4, Table 5): PGD-LS attack on a linear SVM, with budgets of 5 and 25 injected objects, RA is 0.58 and 0.00 respectively, with no indicator triggered.
    • Audio keyword recognition (A.5, Table 6): PGD `2 attack on AUDIO-ConvNet (test accuracy 99%) (n=200, α=5, ε=50), RA 0.00, with no indicator triggered.

    Other ablations and analyses

    • Threshold calibration (A.2, Table 3): gradient-obfuscation models have V(x)>0.4, normal models are close to 0, τ is set to 10% (0.1–0.3 all work); µ for I4 is set to 1%, which the authors call a conservative choice.
    • Scope of F3 (A.6, Table 7): the PGD implementations of Cleverhans 4.0.0, ART 1.11.0, Foolbox 3.3.3, and Torchattacks 3.2.6 all return the endpoint rather than the optimal point.
  5. What are some points that can be further explored?

    The authors state that the difficulty of fully automating M5/M6 is a future direction, and emphasize that indicators not being triggered does not mean robustness; experiments cover more than 15 models across three domains.

    Limitations and future directions noted by the authors

    • Sect. 2.2 (M5): designing for F5 an adaptive loss that covers all components of the defense is "non-trivial step to automate".
    • Sect. 2.2 (M6): the repair for F6 is "not easy to automate".
    • Sect. 2.3: when no indicator is triggered, one cannot assert that no stronger attack exists—as with software testing, only known problems can be fixed.
    • Sect. 5 (Future Work): the future direction is to define repairs that can be fully automated, including how to automatically make attacks adaptive.
    • Sect. 3: after the k-WTA repair, I4 and I6 are still triggered; the authors state that results on some samples can be further improved by increasing the number of steps or the smoothing parameter.

    Experimental coverage

    • The main experiments cover 9 image models (7 defenses + 2 controls) and 6 RobustBench models; the appendix additionally covers 1 each of Windows malware (MalConv), Android (linear SVM), and audio (AUDIO-ConvNet).
    • Each robustness evaluation uses 100 test samples; I1, I2, I6 are computed on a subset of N=10; the paper does not report the number of repetitions or statistical significance tests.
    • Perturbation settings: ε=8/255 for CIFAR models, ε=0.5 for MNIST models; the appendix includes non-lp perturbations (byte embedding space, object injection, spectrograms).
    • Attacks cover PGD, DeepFool, APGDCE, APGDDLR, Extend, PGD-LS; the paper does not report the number of queries or time cost for each attack.
    • The calibration of thresholds τ, µ is based on the V(x), D(x) distributions of the 9 image models (Table 3); the authors state that conservative values were chosen.
  6. Summarize the main content of the paper

    The authors propose six attack failure indicators, IoAF, to automatically debug robustness evaluations; re-evaluation shows that the robustness of most defenses is overestimated.

    An analysis paper on the methodology of adversarial example robustness evaluation: many defenses overestimate robustness because gradient attack optimization fails, and the authors propose a tool to automatically detect such failures.

    • Problem: gradient obfuscation and inappropriate attack hyperparameters cause attacks to fail and defenses to be overestimated; existing evaluation guidelines are difficult to execute systematically.
    • Method: map six categories of failure (F1 gradient shattering, F2 random gradients, F3 implementation error, F4 non-convergence, F5 non-adaptive, F6 unreachable misclassification, of which F3 and F6 are newly proposed by the authors) onto a generalized attack algorithm, and design six binary indicators I1–I6 and fixes M1–M6, forming a semi-automated protocol.
    • Result one: re-evaluated 7 published defenses; after fixes, robust accuracy mostly dropped from 32–95% to 0–3% (e.g., DIST 95%→1%, EN-DV 48%→0%, JPEG-C 74%→0.01%, Table 1); two correctly evaluated control models triggered no indicators.
    • Result two: all 6 models on RobustBench triggered I6, and the authors state their robustness claims may be unreliable (Table 2).
    • Result three: F3 affects the PGD implementations of four libraries: Foolbox, CleverHans, ART, Torchattacks (Table 7); the method also applies to Windows/Android malware and audio domains (Appendix A.3–A.5).
    • Authors' conclusion: IoAF is a first step toward automated, systematic robustness evaluation; limitations are that M5 and M6 are difficult to fully automate, and non-triggering of indicators does not mean no stronger attack exists. The authors have open-sourced the code.

Related papers

Show 3 more papers
Read originalNeurIPS