Gradient Confusion Defense (GCD): Mitigating Score-Based Black-Box Query Attacks via Gradient Confusion

1. 01What problem does this paper attempt to solve?

   The authors aim to weaken score-based black-box query attacks without retraining and without lowering top-1.

   How to weaken score-based black-box query attacks against classifiers that must return confidence scores, without retraining or changing top-1.

   • Deployment scenario: The authors note models are often deployed via remote interfaces like cloud APIs, where callers have only query access and white-box assumptions often fail. Score-based attacks can still estimate gradients from logits or class probabilities and craft adversarial examples within practical budgets, so the authors treat black-box robustness as a deployment requirement.
   • Existing shortcomings: The authors state most existing defenses inject randomness into responses. This can increase variance or slow optimization but does not explicitly control the gradients exposed by outputs; adversaries can average over randomness or increase query budgets to recover stable estimates.
   • Core observation: The authors reframe the problem as what gradients the model should expose to the adversary. The goal is not to add noise but to make estimated gradients or search directions deviate from directions that would reduce the classification margin.
   • Proposed method: Introduce Gradient Confusion Defense (GCD). It is a retraining-free deterministic test-time wrapper that rewrites returned probabilities so that exposed logit gradients align with a direction misaligned from the true adversarial loss gradient, while preserving top-1 and bounded ℓ1 distortion.
   • Threat model:
     • Goal: Generate adversarial examples within an ℓp ball of radius R. Untargeted means prediction is not equal to true label k; the paper also defines targeted. Experiments are only untargeted.
     • Knowledge: Score-based black-box. The abstract says no parameters, no architecture; the introduction says no parameters, no gradients, no training data. Queries return scores or probability vectors, not just labels.
     • Capability: Can query adaptively, up to budget Q. Uses finite differences for zero-order gradient estimation, or random search to minimize the difference between the maximum and second-maximum probabilities.
     • Victim system: A fixed pretrained classifier. GCD does not change its parameters or training, only replaces the probability vector returned to the querier.

   Collapse this answer

2. 02What related research exists?

   Related work is divided into score-based query attacks and randomized or output-manipulation defenses; GCD is distinguished by deterministic gradient obfuscation.

   The paper organizes related work by whether the attack can see scores and whether defense occurs during training or testing.

   Score-based query attacks

   • Zero-order estimation: ZOO by Chen et al. (2017) and Ilyas et al. (2018, 2019) use finite differences or zero-order optimization to estimate gradients from logits or class probabilities.
   • Bandit-TD (Ilyas et al., 2019): Formulates gradient estimation as a bandit problem, exploiting temporal correlation of successive gradients and image spatial structure.
   • Search-based: SquareAttack (Andriushchenko et al., 2020) uses random search with local square perturbations, accepting only proposals that reduce the loss. BruSLeAttack (Vo et al., 2024) targets ℓ0, replacing values on a reduced pixel set with synthetic image values and guiding Bayesian search with historical manipulation results.

   Decision-based queries

   • Labels only: Cheng et al. (2020), Chen and Gu (2020), Chen et al. (2020), Reza et al. (2023) search along decision boundaries. The paper states decision-based attacks often require more queries; Section 3 states score-based attacks are usually stronger and harder to defend. Preliminaries also list transfer-based (Papernot et al., 2017), but experiments do not follow that route.

   Training-time defenses

   • Adversarial training: AT (Tramèr et al., 2018; Salman et al., 2019; Rice et al., 2020) trains on adversarial examples; later additions include data augmentation (Rebuffi et al., 2021), denoising (Salman et al., 2020), and ensembles (Liu et al., 2018). The paper states AT is often costly and may reduce clean accuracy; many such defenses mainly target white-box attacks and can become brittle under gradient masking (Athalye et al., 2018a).

   Output manipulation in query scenarios

   • AAA (Chen et al., 2022): Post-processes returned logits, wraps the original loss with a surrogate loss that has similar degradation but opposite gradient slope, steering optimization in the wrong direction.
   • Random scaling and noise: RLS (Dashtbani et al., 2026) multiplies logits by random coefficients. Boundary defense (Aithal and Li, 2022) adds noise to outputs, RND (Qin et al., 2021) adds noise to inputs, to increase queries needed for successful attacks.
   • Randomization lineage: The introduction also lists Xie et al. (2018), He et al. (2019), Wang et al. (2021), Hung-Quang et al. (2024) among defense literature, stating most introduce randomness that can increase variance or slow optimization but do not explicitly control exposed gradients.

   Baselines and benchmarks: Experimental comparisons are iRND, oRND, RFD, RLS, AAA-sine; attacks are BanditAttack, SquareAttack, SignAttack, BruSLeAttack; data are CIFAR-10 and ImageNet.

   The authors place GCD among retraining-free score-based black-box test-time defenses: using deterministic rewriting to align with a deliberately misaligned gradient direction rather than just adding random perturbations. Section 5.2 further states AAA mainly flips local slopes in output space, while GCD optimizes exposed directions in input space via the Jacobian.

   Collapse this answer

3. 03How does the paper address this problem?

   GCD solves a constrained optimization at test time to align the exposed logit gradient with a direction misaligned from the true adversarial gradient.

   GCD wraps around a fixed classifier: without changing the weights, it changes the returned probabilities from ŷ to ỹ, so that the adversarial loss gradient that the adversary recovers from the output deviates from the true gradient, thereby increasing the queries needed to succeed.

   Adversarial loss and two types of attacks

   • The classifier outputs softmax probabilities. An adversarial example satisfies ‖xadv−x‖p≤R. Untargeted requires the prediction to differ from the true label k.
   • The loss f commonly used by score-based attacks is the largest component of the probability vector minus the second largest. Lowering f weakens the confidence of the current top prediction relative to the strongest competing class.
   • Gradient estimation: finite differences along Gaussian directions give an estimate of the smoothed loss gradient, followed by projected gradient descent. Each step requires querying the target model.
   • Random search: sample directions and accept only when the one-step improvement signal is negative. If the defense flips that sign, it will accept harmful steps or reject useful ones. The adversary actually sees the loss f̃ on the modified probabilities.

   Constrained optimization

   • The obfuscation objective is written as argmin_ỹ⟨∇_x f(x),∇_xf̃(x)⟩. The constraints are that the ℓ1 distance between the original and returned probabilities does not exceed the distortion budget ξ, and that ỹ lies on the probability simplex.
   • The authors give two purposes: a smaller inner product misaligns the estimated gradient and can flip the acceptance sign of random search; the ℓ1 constraint preserves the usability of the scores. They also require the top-1 index to remain unchanged.
   • ỹ appears in both the objective and the gradient, so direct solving has a circular dependency. Because the ℓ1 constraint requires the output to be close, the authors use the local approximation z≈z̃, making the two Jacobians approximately equal, and rewrite the inner product in terms of J(x)=∇x z(x). The authors state that this approximation gives stable improvements in experiments; the main text does not separately report the approximation error.
   • Before optimization, the largest and second-largest classes of ỹ are unknown, so the corresponding indices of ŷ are used to construct ∇z f̃.

   Objective direction and solution

   • Under the above approximation, the logit-gradient objective the authors give is ∇_z^∗f̃(z(x))=−α J(x)^⊤ J(x)∇_z f(z(x)), where α→∞. The authors state that the norm of this vector is not important for GCD's success, and experiments take α=1.
   • This equation is a nonlinear system in ỹ and must also satisfy the ℓ1 and simplex constraints, so it is not solved directly. What is actually solved is: under ℓ1≤ξ, make ∇z f̃ align as much as possible with the above objective direction. After solving, normalize by the sum of components.
   • The solver uses a trust-region interior-point method (Conn et al., 2000), implemented via scipy. If the index of the largest component of ỹ deviates from that of ŷ, the difference between the two is added back to the original top and subtracted from the new top, forcing the argmax to remain unchanged.
   • Experiments take ξ=0.5. On ImageNet, ỹ has large dimension, so optimization is done only on the top-10 and bottom-10 components of the target logit gradient, 20 components total; the rest are fixed to ŷ before normalization. The authors state that these 20 have the largest impact on the objective.

   Theoretical analysis model

   • Section 4.1 uses an analytical objective fdef=λf+(1−λ)f̃, where λ is independently Bernoulli(p) per query. This is not the implementation of Section 4.2; the implementation returns the optimized vector every time, corresponding to the authors' recommended p=0.
   • Assumption 4.1 requires f and f̃ to be Lipschitz; Assumption 4.2 requires the gradient to be Lipschitz. Theorem 4.3 also requires |f−f̃|≤ε and discusses zeroth-order optimization with a sufficiently small smoothing parameter. The authors state that after putting the negative and large-magnitude obfuscation inner product into the bound, the attacker needs more steps T to push the average gradient norm below the threshold; as p→0, (1−p)/p amplifies that term. The derivation is in Appendix B.
   • Theorem 4.4 concerns Gaussian search directions for d≥2: the one-step sign error probability has a lower bound, whose leading term grows with the angle between the two gradients, and the smoothing correction shrinks as the step size μ→0. The derivation is in Appendix C. The authors use this to argue that f̃ should be close to f in function value but misaligned in gradient. Appendix C.3 states that Gaussian directions are used because of rotational invariance and to approximate the sign behavior of isotropic sampling in practice.

   Collapse this answer

4. 04What experiments did the paper conduct?

   GCD's clean accuracy drop is 0% on CIFAR-10 and ImageNet; multiple ASRs are lower than without defense, but on ViT it is not uniformly the lowest.

   Experimental Setup

   • Data and Models: CIFAR-10 uses pretrained ResNet-18; ImageNet uses Torchvision-pretrained ResNet-50 and ViT-B/16. Each dataset randomly samples N=1000 test images.
   • Attacks: Untargeted BanditAttack (ℓ2, ℓ∞), SquareAttack (ℓ2, ℓ∞), SignAttack (ℓ∞). BruSLeAttack (ℓ0) appears in Table 1, not listed in Table 2. Query budget Q=1000.
   • Perturbation Budgets: CIFAR-10 inputs scaled to [0,255], ε=12.75 for ℓ∞ (normalized 0.05), ε=255 for ℓ2 (normalized 1.0). ImageNet inputs in [0,1], ε=0.05 for ℓ∞; ℓ2 follows the original implementation (1275 for Bandit, 5 for Square).
   • Baselines and GCD: iRND adds Gaussian noise σ=0.01 to inputs; oRND adds Gaussian noise σ=1.0 to output logits; RFD adds noise to intermediate features; RLS multiplies logits by a uniform coefficient on [0.5,10]; AAA only tests AAA-sine (temperature 1, deviation 0.5, attractor interval 4, calibration weight 5, learning rate 0.1, 100 inner iterations). The authors cite Chen et al. (2022) that AAA-linear is vulnerable to adaptive attacks. GCD uses ξ=0.5, and ImageNet only optimizes 20 components.
   • Metrics: ASR is the proportion of samples successfully attacked within budget; Average Queries is the average number of queries for successful attacks; Clean Accuracy Drop is also reported. The paper does not mention LLM review, human agreement, or number of repetitions.
   • Adaptive: Table 4 puts EOT and Pendulum into Square Attack, k=10, evaluating 1000 ImageNet images with ViT-B/16. Max takes the higher ASR of the two variants.

   Main Results

   The table below shows ASR, without defense and with GCD. The CIFAR-10 ResNet-18 row is from Table 1, and the two ImageNet rows are from Table 2. Q=1000, untargeted. Average queries and BrusLe-ℓ0 are not included in the table.

   Swipe horizontally to read the table

   Model and Defense	Bandit-ℓ2	Bandit-ℓ∞	Square-ℓ2	Square-ℓ∞	Sign-ℓ∞
   ResNet-18 no defense	82.68%	98.03%	87.76%	99.38%	98.97%
   ResNet-18 GCD	0.94%	28.12%	4.15%	30.71%	14.53%
   ResNet-50 no defense	46.73%	87.05%	81.45%	99.47%	91.33%
   ResNet-50 GCD	3.34%	35.65%	13.22%	47.94%	20.17%
   ViT-B/16 no defense	38.17%	76.46%	70.64%	97.90%	80.05%
   ViT-B/16 GCD	3.35%	21.94%	12.38%	33.83%	21.44%

   • Relative to the no-defense rows in the same table, GCD has lower ASR in all cases, and GCD's Acc. Drop is 0% in Tables 1–2. The authors claim this suppresses attack success rate while preserving predictions, and in the abstract and Section 6 claim state-of-the-art.
   • The authors claim GCD outperforms almost all baselines on non-transformers. Counterexamples outside the table: ResNet-18 Bandit-ℓ∞, RLS is 25.11%, GCD is 28.12%; ResNet-50 Square-ℓ∞, AAA is 47.06%, GCD is 47.94%.
   • On ViT-B/16 Square-ℓ2, Square-ℓ∞, Sign-ℓ∞, lower ASR comes from RLS (10.29%, 30.36%, 13.39%) or AAA (10.9%, 26.74%, 15.1%), not GCD. BrusLe-ℓ0 only appears in Table 1: no defense 79.78%, GCD 38.59%.

   Compared with Other Defenses and Output Shift

   • Accuracy Drop: RLS is 0% across all three rows. AAA is 0.1%, 0.3%, 0% on ResNet-18, ResNet-50, ViT-B/16. iRND is 1%, 0.2%, 0.2%; oRND is 0.6%, 1.7%, 1.8% (Tables 1–2). The authors claim iRND and oRND trade accuracy for robustness and provide weaker protection; AAA and RLS preserve accuracy but are more vulnerable to adaptive attacks.
   • ℓ1: Table 3 compares ŷ and the defended output on 1000 ImageNet images, without specifying the architecture. GCD is min 0.07, avg 0.58, max 1.25. iRND, oRND, RLS, AAA have avg 0.11, 0.26, 0.33, 0.28. The authors claim GCD has a larger shift, consistent with shaping scores to misalign gradients rather than just adding noise.
   • Average Queries: This metric averages only over successful samples. In Table 1, GCD is 20.95 for Square-ℓ2, and no defense is 466.29. A smaller mean cannot be directly read as failed samples also stopping earlier.

   Gradient Angles

   • Figure 1: 10 ImageNet test images, each color corresponds to 200 square-ℓ∞ iterations; dashed lines and numbers are per-sample average angles. The figure text does not specify the architecture. GCD annotations include 24.5°, 29.9°, 36.0°, 46.8°, 47.4°, 49.0°, 49.4°, 51.7°, 68.9°, 84.4°. The authors claim many are close to about 50° and call this a systematic bias. AAA annotations are 0.0°; the authors claim it almost exposes the true direction.
   • RLS: The figure shows 0.0°, 0.5°, and 55.4°, 55.9°, 58.4°, 66.5°. The authors claim there are also angles close to 90° and speculate that some probabilities are mapped to one-hot, making the post-defense gradient near 0. Figure 1's text annotations do not show 90°.
   • Figure 2: 10 CIFAR-10 images, 100 Square Attack iterations. The authors claim AAA's average angle quickly falls to about 8°–10°, GCD stays around 25°–28°, with many samples at 30°–45°. The authors claim the difference lies in whether misalignment can be maintained along the entire trajectory, and claim this is consistent with Theorem 4.4. The CIFAR-10 model in the main text is ResNet-18; the caption does not specify another architecture.

   Adaptive Attacks

   • EOT and Pendulum are added to Square Attack, k=10, ViT-B/16, 1000 ImageNet images. EOT averages over k randomized outputs; PendulumAttack iteratively estimates and exploits the defense pattern. The paper does not extend this protocol to the other two models.
   • Table 4 ASR/average queries: AAA is EOT 21.94%/85.44, Pen-Square 40.53%/306.08, Max 40.53%; RLS is 36.19%/215.83, 23.55%/85.44, Max 36.19%; GCD is 24.04%/159.69, 30.12%/209.33, Max 30.12%.
   • The authors treat Max as the main result and claim that under the setting of "taking the more successful of the two variants," GCD is more robust to adaptive attacks than AAA and RLS. By variant, AAA has lower ASR under EOT, and RLS is lower under Pen-Square. The authors claim Figure 2 helps explain this comparison but does not by itself establish causation: averaging targets variance and does not remove deterministic directional bias; Pendulum can exploit AAA's oscillatory structure.

   Other Ablations and Analyses

   • The paper does not report ASR comparisons for ξ, p, or ImageNet's full 1000 classes versus the 20-component subset.
   • RFD is written into the experimental setup, but Tables 1–2 do not give ASR or Acc. Drop for it.
   • Table 1: iRND's BrusLe-ℓ0 ASR is 82.9%, higher than no defense's 79.78%.
   • Table 2: oRND's Bandit-ℓ2 ASR on ResNet-50 is 51.47%, higher than no defense's 46.73%.
   • Table 2: oRND's Sign-ℓ∞ ASR on ViT-B/16 is 81.44%, higher than no defense's 80.05%.
   • The authors claim equations (7)(15)(16) optimize the input-space direction after composition with the Jacobian, a step AAA does not take.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors only note that the Jacobian adds about 15% inference overhead and plan to explore alternative optimization.

   Limitations and future directions noted by the authors

   • Jacobian overhead: Section 6 states that there is a limitation to discuss: GCD requires computing the Jacobian, which adds computational overhead, increasing inference time by approximately 15%.
   • Future directions: The same section states that future work could explore alternative optimization methods and approaches that bypass this obstacle.
   • The paper does not have a separate Limitations subsection. Apart from this, the main text does not explicitly frame any other result as a limitation or future work.

   Experimental coverage

   • Models and data: CIFAR-10 uses a pretrained ResNet-18; ImageNet uses Torchvision-pretrained ResNet-50 and ViT-B/16. N=1000 test images are randomly sampled per dataset (Section 5.1).
   • Main protocol: untargeted, Q=1000. Tables 1–2 include Bandit-ℓ2/ℓ∞, Square-ℓ2/ℓ∞, Sign-ℓ∞; BrusLe-ℓ0 appears only in Table 1. Perturbation budgets follow ε in Section 5.1. The utility metric is Clean Accuracy Drop; the paper does not report absolute clean accuracy.
   • Comparison defenses: the tables include iRND, oRND, RLS, AAA-sine, and GCD. The setup text also lists RFD, but the tables do not give its ASR or accuracy drop.
   • Adaptive attacks and angle figures: Table 4 uses ViT-B/16, 1000 ImageNet images, k=10 for EOT and Pen-Square. Figure 1 uses 10 ImageNet images, 200 square-ℓ∞ iterations per sample, with no architecture stated. Figure 2 uses 10 CIFAR-10 images, 100 Square Attack iterations.
   • Quantities not reported in the paper: number of repetitions, targeted ASR, ASR for different ξ or different p, and ASR for full ImageNet optimization versus the 20-component subset. The Bernoulli(p) switching of Section 4.1 has no corresponding experimental sweep. Related work discusses decision-based attacks, but the main text does not give their ASR against GCD.

   Collapse this answer

6. 06Summarize the main content of the paper

   GCD uses Jacobian-constrained rewriting of returned probabilities to reduce the ASR of various score-based black-box attacks with zero clean accuracy drop.

   GCD is a test-time defense against score-based black-box query attacks: the classifier weights are unchanged, and only the probability vector returned to the querier is modified.

   • Problem: Remote interfaces typically provide only queries and confidence scores. The authors argue that merely adding randomness to the output does not stop an adversary who increases the budget or averages over the randomness, so they aim to actively steer the exposed gradient away from the true adversarial loss gradient.
   • Method: Within an ℓ1 budget while preserving top-1, they construct a misaligned direction using the Jacobian of the logits, then solve for the returned probabilities with a trust-region interior-point method. Experiments use ξ=0.5; ImageNet optimizes only 20 components. The random switching used in the analysis is not the implementation itself; the authors recommend always returning the modified scores. Theorem 4.3 and 4.4 discuss the number of steps needed for zeroth-order optimization and the one-step sign error of random search, respectively.
   • Main results: CIFAR-10 (ResNet-18) and ImageNet (ResNet-50, ViT-B/16), 1000 images each, Q=1000, untargeted. GCD's clean accuracy drop is 0% in all cases. On ResNet-18, Square-ℓ2 ASR drops from 87.76% to 4.15%, and Sign-ℓ∞ from 98.97% to 14.53% (Table 1). On ResNet-50, GCD gives 3.34% for Bandit-ℓ2 and 47.94% for Square-ℓ∞ (Table 2). On ViT, Square-ℓ∞ drops from 97.90% to 33.83%, but the ASRs for Square-ℓ2, Square-ℓ∞, and Sign-ℓ∞ are higher than those of RLS or AAA.
   • Adaptive attacks and conclusions: On ViT-B/16, taking the higher ASR between EOT and Pendulum, GCD is 30.12%, AAA is 40.53%, and RLS is 36.19% (Table 4). The authors state that GCD uses deterministic directional misalignment against score-based query attacks and call it state-of-the-art; Section 6 also states that the Jacobian increases inference time by approximately 15%.

   Collapse this answer

Related papers