Robustness-optimization defenses have shared vulnerabilities; breaking one can expose the entire defense family

The paper finds that adversarial robustness-optimization defenses have shared vulnerabilities, and breaking one representative defense may expose defenses of the same type together. The authors propose a stricter transfer-only testing protocol and a simple adaptive attack, PGDTransfer, to distinguish genuine transferability from performance degradation caused by distortion, and introduce Adversarial Sensitivity Maps to visualize and quantify shared alignment, covering stochastic and non-differentiable defenses. On adversarially trained classifiers, purification-based defenses, and LVLMs with robust vision encoders, the authors find natural transferability within each robustness family, i.e., transfer is possible even with simple PGD-style optimization. The risk for purification-type defenses is already severe: at ε=4/255, PGDTransfer achieves an average transfer attack success rate of 80.4% against three types of purifiers: filtering, compression, and diffusion. The authors accordingly recommend that future defenses treat vulnerability diversity and transfer-only isolation as safety objectives, rather than optimizing only the robustness of a single defense.