Uncovering a Flawed Adversarial Example Defense: Fixing One Line of Code Breaks Sabre

1. 01What problem does this paper attempt to solve?

   Verify the robustness claims of the IEEE S&P 2024 defense Sabre, point out flaws in its evaluation, and break the defense four times by modifying code.

   Verify whether the robustness claims of the adversarial example defense Sabre hold. Sabre is a defense accepted at IEEE S&P 2024 that claims to reduce the best attack success rate on CIFAR-10 from 29% to 11%, but the authors find that its evaluation contains mathematically impossible results and clear signs of gradient masking.

   • Problem setting: Evaluation of adversarial example defenses needs to follow standard best practices, but Sabre claims 92.66% accuracy at ε=0.5 (theoretically it should be close to random guessing), and accuracy under attack is instead higher than without attack.
   • Shortcomings of existing methods: Sabre did not test adaptive attacks, incorrectly implemented the adversarial training baseline (reporting 70% success rate on MNIST instead of the correct 11%), and compared with the known-broken ME-Net without considering the breaking method.
   • Core observation: Sabre's preprocessing defense has gradient masking, its claimed end-to-end differentiability does not match the actual implementation, and there are bugs in the evaluation code.
   • Threat model: The authors act as attackers, using an adaptive attack, fully knowing the defense implementation details (white-box), bypassing the defense by modifying the defense code or attack hyperparameters; the victim system is the Sabre-defended image classifier (MNIST and CIFAR-10).
   • Paper contribution: By modifying 1 line of code, they break Sabre and its subsequent fixed versions four times, reducing robust accuracy to 0% or 4%.

   Collapse this answer

2. 02What related research exists?

   The paper involves related work on adversarial-example defense evaluation standards, adaptive attacks, gradient masking, and preprocessor defenses.

   Adversarial-example defense evaluation standards

   • Carlini et al. [3] — Proposed a checklist of best practices for evaluating adversarial robustness, including verifying that iterative attacks outperform single-step attacks, conducting adaptive attacks, and attempting gradient-free attacks. This paper uses this checklist as the basis for criticizing Sabre's evaluation flaws.
   • Tramèr et al. [7] — Discussed adaptive attacks as the de facto standard for evaluating defenses, pointing out that defenses against non-adaptive attacks have limited utility.

   Gradient masking and attack methods

   • Athalye et al. [1] — Proposed the BPDA (Backward Pass Differentiable Approximation) method for gradient estimation of non-differentiable defenses. This paper points out that Sabre incorrectly uses BPDA for an already differentiable defense, thereby masking the true gradient.
   • Croce and Hein [4] — Proposed AutoAttack, an evaluation method integrating multiple parameter-free attacks. The Sabre authors mistakenly regard AutoAttack as an adaptive attack, whereas this paper points out that AutoAttack is not a substitute for adaptive attacks.

   Defense baselines compared

   • Madry et al. [6] — Adversarial Training, the current state-of-the-art defense method. Sabre incorrectly implements this method, reporting a 70% attack success rate on MNIST instead of the correct 11%.
   • Yang et al. [9] — ME-Net, a preprocessor defense that was broken two months after its 2019 publication. Sabre compares with it but does not account for the breaking method, and does not mention in the paper the fact that ME-Net has been broken.
   • Xu et al. [8] — Feature Squeezing, an input-discretization defense proposed in 2017. This paper cites it as a baseline in the second attack, pointing out that Sabre's binarization defense success rate (21%) is instead higher than plain binarization (13%).

   Baselines and benchmarks

   • Baseline methods: adversarial training (Madry et al.), ME-Net, Feature Squeezing.
   • Benchmark datasets: MNIST, CIFAR-10.

   The authors position the distinction of this paper from related work as: not only pointing out evaluation flaws, but also demonstrating the defense's vulnerability through actual code modifications, and emphasizing the necessity of adaptive attacks.

   Collapse this answer

3. 03How does the paper address this problem?

   Through code auditing and four adaptive attacks (modifying 1 line of code), the authors progressively break Sabre and its three fixed versions.

   Overall approach: The authors first used GPT-4 to help identify mathematical contradictions in the original paper's evaluation, then conducted four adaptive attacks against the defense code, each modifying only 1 line of code (removing a function call, inserting a function call, commenting out code, changing the sign of a hyperparameter), reducing robust accuracy to 0% or 4%.

   Identification of evaluation flaws

   • GPT-4-assisted analysis: The evaluation checklist [3] and screenshots of the Sabre paper were input into GPT-4, which identified two mathematically impossible results: 92.66% accuracy at ε=0.5 (which should be close to random guessing), and accuracy under attack (95.68%) higher than the no-attack state (83.57%).
   • Standard compliance check: Against [3], it was pointed out that Sabre did not verify that iterative attacks outperform single-step attacks, did not conduct adaptive attacks, did not attempt gradient-free or hard-label attacks, did not conduct random noise attacks, and did not perform ablation studies.

   First attack: Removing the redundant BPDA wrapper

   • Problem localization: Sabre claimed to be end-to-end differentiable, but the evaluation code wrapped the preprocessing step in BPDA, replacing the true gradient with the estimated gradient f'(x)=x, resulting in gradient masking.
   • Attack method: Delete the BPDAWrapper wrapper and restore true gradient computation.
   • Key parameters: ε=0.3 on MNIST, ε=8/255 on CIFAR-10, PGD iterations increased from 7 steps to 100 steps.

   Second attack: Inserting BPDA into the discretization function

   • Defense change: The authors introduced a diff_round differentiable discretization function (MNIST binarized to {0,1}, CIFAR-10 discretized to one decimal place), but the implementation had a bug, with gradients almost everywhere negative (Figure 1).
   • Attack method: Insert a BPDA wrapper into diff_round to bypass the incorrect gradient.
   • Key finding: Attack success rate on MNIST was 21%, higher than the 13% of plain binarization [8]; success rate on CIFAR-10 was 100%.

   Third attack: Commenting out the discretization code

   • Defense change: The authors released a 300-line code modification, adding hard discretization (discretizing the input into 8 levels) to the DenoisingCNN forward pass, causing gradients to be identically zero.
   • Attack method: Comment out the discretization code lines during attack and restore them during evaluation, equivalent to a BPDA attack.
   • Key parameters: Against the officially provided CIFAR-10 pretrained model.

   Fourth attack: Negating the PGD step size

   • Defense change: The authors removed all discretization steps, claiming "reconstruction without discretization", but the code could not reproduce the original results (accuracy after PGD attack on CIFAR-10 reached 100%, inconsistent with the 93% claimed in the paper).
   • Attack method: Change the step size hyperparameter α of EoT-PGD from 2/255 to -2/255, i.e., update in the opposite direction of the gradient.
   • Phenomenon: Defense accuracy dropped to 4%, below the random guessing baseline.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The authors broke Sabre four times on MNIST and CIFAR-10, reducing robust accuracy to 0%, 21%/100%, 0%, and 4%, respectively.

   Experimental setup

   • Defense under test: The original version of Sabre and three subsequent fixed versions (introducing discretization, hard discretization, and reconstruction without discretization, respectively).
   • Datasets: MNIST (perturbation ε=0.3), CIFAR-10 (perturbation ε=8/255).
   • Attack methods: PGD (7 steps or 100 steps), EoT-PGD (7 steps, eot_steps=10).
   • Baselines: Adversarial training [6], Feature Squeezing [8] (MNIST binarization comparison only).
   • Metrics: Robust accuracy, attack success rate.
   • Review method: GPT-4 was used to check evaluation flaws in the original paper (Section 3); the authors manually verified the accuracy of GPT-4's identifications.

   Main results

   Swipe horizontally to read the table

   Attack round	Defense version	Dataset	Modification	Robust accuracy	Source
   First	Original version	MNIST	Remove BPDA wrapper	0%	Section 6.1
   First	Original version	CIFAR-10	Remove BPDA wrapper	0%	Section 6.1
   Second	First fix (discretization)	MNIST	Insert BPDA wrapper	21% attack success rate	Section 6.3
   Second	First fix (discretization)	CIFAR-10	Insert BPDA wrapper	100% attack success rate	Section 6.3
   Third	Second fix (hard discretization)	CIFAR-10	Comment out discretization code	0%	Section 6.4
   Fourth	Third fix (no discretization)	CIFAR-10	Negate PGD step size	4%	Section 6.5

   • Authors' interpretation: The first attack showed that the BPDA wrapper in the original evaluation was the source of gradient masking; the second attack showed that the introduced discretization function was implemented incorrectly (gradients negative); the third attack showed that hard discretization caused gradients to be zero; the fourth attack showed that the model's gradient direction was abnormal.

   Evaluation of the defense

   • What was tested: Whether the robustness claimed in the original Sabre paper holds.
   • Results: The original evaluation contained mathematical contradictions (92.66% accuracy at ε=0.5) and did not conduct adaptive attacks. The authors reduced robust accuracy to 0% or 4% through four code modifications.
   • Authors' interpretation: Sabre's robustness claims stem from evaluation bugs and gradient masking, not genuine robustness.

   Transfer and generalization

   • What was tested: In the second attack, BPDA was inserted into the MNIST binarization defense and the CIFAR-10 one-decimal-place discretization defense, respectively.
   • Results: Attack success rate on MNIST was 21%, higher than the 13% of the Feature Squeezing baseline; success rate on CIFAR-10 was 100%.
   • Authors' interpretation: Sabre's discretization defense actually reduced robustness, below the simple binarization baseline.

   Mechanism analysis

   • What was tested: The gradient properties of the diff_round function (Figure 1).
   • Results: The function's gradient is almost everywhere negative in the input range 0.0 to 0.3, causing gradient descent to move away from adversarial examples.
   • Authors' interpretation: This is the most extreme form of gradient masking the authors have seen; the gradient not only hides the direction but points in the wrong direction.

   Other ablations and analyses

   • Number of iterations: In the first attack, PGD on MNIST increased from 7 steps to 100 steps, and accuracy dropped from <5% to 0%.
   • Discretization precision: In the second attack, MNIST changed from 0 decimal places (binarization) to 1 decimal place, and the attack success rate rose from 21% to 100%.
   • Model source: The CIFAR-10 results in the second attack are based on a model trained by the authors themselves (using the original authors' code), not a model provided by the original authors.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out that Sabre did not conduct adaptive attacks and did not follow standard evaluation practices, and that the cause of the phenomenon in the fourth attack is not yet clear.

   Limitations and future directions noted by the authors

   • Lack of adaptive attacks: Sabre did not test adaptive attacks; the authors cite [3] in pointing out that "defending against non-adaptive attacks is necessary but not sufficient" (Section 5).
   • Evaluation practice flaws: Sabre did not follow standard best practices, including not verifying that iterative attacks outperform single-step attacks, not attempting gradient-free attacks and hard-label attacks, not conducting random noise attacks, and not performing ablation studies (Section 4).
   • Baseline implementation errors: Sabre incorrectly implemented the adversarial training baseline and, when comparing with the known-broken ME-Net, did not consider the breaking method (Section 5).
   • Code reproducibility: In the fourth attack, the authors found that the latest code could not reproduce the original paper's results (accuracy after PGD attack on CIFAR-10 was 100% rather than the 93% claimed in the paper), and the authors could not run the MNIST model (clean accuracy dropped to 10%) (Section 6.5).
   • Unexplained phenomenon: In the fourth attack, negating the step size caused accuracy to drop to 4%; the authors state that "explaining this phenomenon is beyond the scope of this paper" and that it may be due to model retraining causing reversed gradients or changed behavior of the reconstructed model (Section 6.5).

   Scope of experimental coverage

   • Models under test: Four versions of the Sabre defense (original and three fixed versions), based on the MNIST and CIFAR-10 datasets.
   • Attack methods: PGD (7 steps, 100 steps), EoT-PGD (7 steps, eot_steps=10).
   • Review model: GPT-4 was used to identify evaluation flaws in the original paper (Section 3).
   • Sample size: The paper does not report a specific sample size.
   • Number of repetitions: The paper does not report the number of repeated experiments.
   • Not covered: The paper does not report tests on other datasets in the original Sabre paper (such as SPEECH-COMMANDS); it does not report measurements of the defense's computational overhead.

   Collapse this answer

6. 06Summarize the main content of the paper

   Through four adaptive attacks each modifying 1 line of code, the authors reduced the robust accuracy of the IEEE S&P 2024 defense Sabre to 0% or 4%, pointing out that its evaluation has flaws.

   This paper targets Sabre, an adversarial example defense accepted at IEEE S&P 2024, points out that its evaluation contains mathematically impossible results and signs of gradient masking, and verifies its fragility through four adaptive attacks.

   • Problem: Sabre claims to reduce the attack success rate on CIFAR-10 from 29% to 11%, but the authors find that its evaluation violates standard practices and that its results are mathematically impossible (e.g., 92.66% accuracy at ε=0.5).
   • Method: the authors first use GPT-4 to identify evaluation flaws, then carry out four attacks against the defense code: removing a redundant BPDA wrapper, inserting BPDA for the incorrect discretization function, commenting out the hard discretization code, and negating the PGD step-size hyperparameter.
   • Results:
     • First attack: robust accuracy drops to 0% on MNIST and CIFAR-10 (Section 6.1).
     • Second attack: attack success rate 21% on MNIST (above the 13% baseline) and 100% on CIFAR-10 (Section 6.3).
     • Third attack: robust accuracy of the official CIFAR-10 model drops to 0% (Section 6.4).
     • Fourth attack: robust accuracy on CIFAR-10 drops to 4% (Section 6.5).
   • Conclusion: the authors argue that Sabre's robustness claims stem from evaluation bugs and gradient masking, emphasize the importance of adaptive attacks and standard evaluation practices, and point out that the community needs stricter scrutiny of defense papers to avoid similar errors.

   Collapse this answer

Related papers