Ensemble Everything Everywhere: Multi-Scale Aggregation for Adversarial Robustness

1. 01What problem does this paper attempt to solve?

   Addressing the adversarial vulnerability of image classifiers, the authors claim it approaches or exceeds RobustBench SOTA without adversarial training.

   Core problem: image classifiers have adversarial vulnerability—perturbations imperceptible to the human eye can cause the model to misclassify, which the authors consider a manifestation of the inconsistency between the machine vision and human vision classification functions.

   • Setting and motivation: The authors state that adversarial examples are ubiquitous and can transfer between models, yet do not transfer to humans, threatening the robustness, reliability and alignment of models; the authors take aligning human and machine vision functions as the goal, with adversarial robustness as the main case.
   • Shortcomings of existing methods: The authors believe robustness improvements have historically come mainly from adversarial training (brute force), with little attention to improving the model backbone and training paradigm themselves.
   • Core hypothesis: The authors propose the Interpretability-Robustness Hypothesis—models whose adversarial attacks appear human-understandable will also be more adversarially robust; and hypothesize that bio-inspired designs such as multi-resolution, jitter and noise can narrow the human-machine function gap.
   • Paper's approach: Proposes multi-resolution input stacking and the CrossMax robust ensemble (a self-ensemble that aggregates intermediate-layer predictions), and states that robustness is obtained without adversarial training or additional data; byproducts are turning the classifier and CLIP into controllable image generators and constructing transferable attacks on vision-language models.
   • Threat model:
     • Attacker goal: Add L∞-norm-bounded perturbations to images (standard 8/255) to make the classifier misclassify as a target class or any wrong class.
     • Knowledge and capability: White-box—attacks optimize against model gradients; evaluation uses the four RobustBench AutoAttack attacks, including the rand version for randomized models.
     • Victim systems: ImageNet-pretrained ResNet18/ResNet152 fine-tuned classifiers on CIFAR-10/100, and ResNet18 trained from scratch; commercial vision-language models such as GPT-4 are also tested.

   Collapse this answer

2. 02What related research exists?

   The paper positions its work along four lines: adversarial examples, ensemble uncertainty, biomimetic vision, and multi-resolution representations.

   Adversarial examples and adversarial training

   • Szegedy et al. (2013)—first described the adversarial example phenomenon; Goodfellow et al. (2015)—explained adversarial examples and proposed FGSM, which is used in this paper's adversarial fine-tuning to generate attack samples.
   • Chakraborty et al. (2018)—survey of adversarial attacks and defenses; the authors state that most past robustness improvements came from adversarial training.
   • Croce et al. (2020) and Croce and Hein (2020)—the RobustBench benchmark and AutoAttack attack suite, used in this paper for evaluation and comparison with the leaderboard.

   Ensembles and uncertainty

   • Lakshminarayanan et al. (2017), Ovadia et al. (2019)—deep ensembles averaging logits or probabilities improve accuracy and uncertainty; the paper points out that such aggregation can be dominated by a single anomalous prediction, creating a single point of failure.
   • Ashukha et al. (2021)—pointed out pitfalls in ensemble uncertainty estimation; Wu and Gales (2021)—discussed calibration of ensemble members.
   • Wilson (1977)'s Vickrey auction and Azar et al. (1999), Mitzenmacher et al. (2001) on load balancing—the inspiration for CrossMax using the k-th highest prediction to suppress overconfident members.

   Biomimetic vision and multi-resolution representations

   • Martinez-Conde et al. (2004), Wandell (1995)—human eye microsaccades and retinal resolution variation, the biomimetic basis for this paper's multi-resolution plus jitter design.
   • Burt and Adelson (1983)—Gaussian pyramid multi-resolution representation; Havasi et al. (2021), Fort et al. (2021b)—inputting multiple sub-images simultaneously or the same image with multiple augmentations, which this paper states goes further by inputting multiple augmentations simultaneously as channels.

   Intermediate-layer representations and related observations

   • Olah et al. (2017), Carter et al. (2019)—feature visualization shows networks build understanding layer by layer from edges to objects, and this paper uses this to ask about the intermediate-layer representations of adversarial images.
   • Schaeffer et al. (2024)—study of transferability of universal image jailbreaks across vision-language models; the authors state that this paper's transferable attacks are constructive evidence to the contrary.

   Baselines and benchmarks: The comparison baselines are the RobustBench leaderboard SOTA (CIFAR-10's [39] 73.71%, CIFAR-100's [40] 42.67%, Table 1/2), and Wang et al. (2023)'s CIFAR-100 adversarially trained model (attack visualization comparison in Figure 12); the benchmarks are CIFAR-10/100 plus RobustBench AutoAttack. The authors' positioning: without retuning the backbone or relying on adversarial training, use a multi-resolution prior plus robust ensembling to give the model robustness by default.

   Collapse this answer

3. 03How does the paper address this problem?

   Multi-resolution channel-stacked input plus CrossMax robust aggregation of intermediate-layer predictions forms a self-ensemble classifier that requires no adversarial training.

   Overall idea: randomly expand the input image into a channel stack of multiple resolutions (biomimetic saccades), then use the Vickrey-auction-inspired CrossMax algorithm to aggregate intermediate-layer predictions into a self-ensemble, making it hard for an attacker to fool all layers at once.

   Multi-resolution input

   • Turn an R×R×3 image into N versions at different resolutions (for CIFAR, ρ={32,16,8,4}), resample each back to R×R, and stack them by channel into an R×R×(3N) input.
   • Add random noise (standard deviation 0.2/1.0), ±3 jitter in the x-y plane, random contrast changes, and random color-to-grayscale shifts to each version; the authors state this also compresses the dimensionality of the input space available to the attacker.
   • Only two structural changes: the first convolutional layer's input channels change from 3 to 3N, and the final linear layer is changed to the corresponding number of classes; both layers are randomly initialized.

   CrossMax robust ensemble

   • For logits Z of shape [B, N, C], first compute Ẑ = Z − max(Z, axis=2) (subtract each predictor's maximum to prevent a single predictor from dominating), then subtract the per-class maximum across predictors (to prevent a class from winning via an outlier), and finally take the median by class (equivalent to model ensembling) or top-k (self-ensemble).
   • Inspired by the Vickrey auction: use the k-th highest prediction to constrain erroneous but overconfident members; the authors state this makes the attacker face a dynamically changing k-th highest prediction, forming a cat-and-mouse game.
   • Ablation: the two normalization steps A (subtract predictor maximum) and B (subtract class maximum) work best in the order AB; at 4/255 the robust accuracy of each combination is {_, A, B, AB, BA} = {4, 4, 0, 22, 0}%.

   Intermediate-layer self-ensemble

   • Train linear probes on the intermediate-layer activations of a frozen backbone (1 epoch per layer) to obtain L sets of classification logits, aggregated with CrossMax (top3) into a self-ensemble.
   • The basis is what the authors call adversarial layer de-correlation: attacks designed for a certain layer mainly affect the vicinity of that layer, and attacks do not transfer between early/middle/late layers (Figure 6, 21).

   Training configuration

   • Fine-tuning: ImageNet-pretrained ResNet18/152, Adam, constant learning rate 3.3×10⁻⁵ (the authors state the small learning rate is key), fine-tuned on CIFAR-10/100; Table 1 is 20 epochs (the main text separately states 10 epochs plus 3 epochs at 3.3×10⁻⁶).
   • Training from scratch: ResNet18 with batch norm off, mixup augmentation, lr 10⁻³ for 380 epochs then 10⁻⁴ for 120 epochs, batch 512.
   • Adversarial fine-tuning: each batch uses single-step FGSM (L∞=8/255) to generate attack samples online for training.

   Evaluation method

   • Use RobustBench AutoAttack (APGD-CE, APGD-T, FAB-T, SQUARE), and finally the rand version for randomized models (APGD-CE followed by APGD-DLR), L∞=8/255.
   • Quick evaluation on 128 test images, detailed evaluation on 512 or 1024; the rand version is run only at the end on 128 images, without targeted tuning.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On RobustBench AutoAttack (rand, L∞=8/255), without adversarial training CIFAR-10 reaches 71.88% and CIFAR-100 reaches 48.16%.

   Experimental setup

   • Models tested: ImageNet-pretrained ResNet152 and ResNet18 fine-tuned models, ResNet18 trained from scratch, ViT-B/16 (Figure 25); transfer attacks test GPT-4, Bing Copilot, Claude 3 Opus, Gemini Advanced (Tables 3, 4).
   • Datasets: CIFAR-10 and CIFAR-100 (50,000 32×32 training images each).
   • Baselines: RobustBench leaderboard SOTA—CIFAR-10 [39] 73.71%, CIFAR-100 [40] 42.67%; standard mean/median ensembles and single models (Figure 4).
   • Metrics: adversarial accuracy (adv acc) and clean test accuracy (test acc), with attacks being the AutoAttack rand version (APGD-CE→APGD-DLR), L∞=8/255.
   • Sample sizes: most results use 128 test images, CIFAR-100 self-ensembles use 512 (with ± intervals), ResNet18 trained from scratch uses 1024; the paper does not report the number of repetitions.

   Main results

   According to Table 1 (no adversarial training) and Table 2 (lightweight adversarial training), rand AutoAttack L∞=8/255:

   Swipe horizontally to read the table

   Dataset	Adversarial training	Method	#Samples	Test acc	Adv acc
   CIFAR-10	×	ResNet18* self-ensemble (trained from scratch)	1024	76.94	64.06
   CIFAR-10	×	ResNet152 multi-resolution backbone	128	89.17	41.44
   CIFAR-10	×	ResNet152 self-ensemble	128	87.14	53.12
   CIFAR-10	×	ResNet152 3-ensemble of self-ensembles	128	90.20	71.88
   CIFAR-10	✓	ResNet152 3-ensemble of self-ensembles	128	87.00	78.13
   CIFAR-100	×	ResNet152 multi-resolution backbone	128	65.70	25.00
   CIFAR-100	×	ResNet152 3-ensemble of self-ensembles	512	67.71	48.16±2.65
   CIFAR-100	✓	ResNet152 3-ensemble of self-ensembles	512	61.17	51.28±1.95

   • The authors state that without adversarial training the CIFAR-10 result ranks 3rd on RobustBench, and CIFAR-100 exceeds the then-best (42.67%) by about +5%.
   • The authors state that with lightweight adversarial training, CIFAR-10 exceeds SOTA by about 5% and CIFAR-100 by about 9%, with larger gains on the harder CIFAR-100.
   • The authors state that robustness stacks layer by layer: multi-resolution input about 40%, adding self-ensemble about +10%, and ensembling 3 self-ensembles about +20% (Figure 7).

   Ensemble method comparison (Figure 4)

   • What was tested: 10 ResNet18 models with different aggregation methods, robust accuracy under progressively stronger L∞ attacks (APGD-T, APGD-CE).
   • Results: mean/median ensembles drop to about 0% at 8/255; CrossMax still has >0 robust accuracy at 8/255, and at 6/255 is 17 times the standard ensemble on CIFAR-10 and 12 times on CIFAR-100 (annotated in the figure).
   • Authors' interpretation: CrossMax can bring non-trivial robustness gains to an ensemble of individually fragile models.

   Intermediate-layer decorrelation and self-ensembles (Figures 5, 6, 25)

   • What was tested: linear probes measure the accuracy and target-class probability of each intermediate layer under attacks targeting the final layer (8/255, 128 attacks); and the effect of attacks targeting layer α on layer β (512 attacks).
   • Results: intermediate-layer accuracy on attacked images drops only slightly, and target-class probability rises only in the last few layers; attack effects are localized by early/middle/late layers and do not transfer across regions.
   • Authors' interpretation: intermediate-layer representations are not fooled by attacks targeting the overall classifier and can serve as proactive defense; Figure 25 shows self-ensembles still have non-zero robust accuracy at 5/255, while mean ensembles and the final layer fail at about 3/255.

   Interpretability and transfer attacks

   • Gradient attacks on multi-resolution models (400-step SGD, η=1, allowed up to 128/255), the authors state the perturbations exhibit human-understandable semantic changes (Figures 9, 10, 22, 23); recognizable images can be generated from a uniform gray image (Figures 11, 24).
   • Multi-resolution attacks transfer to commercial vision-language models: at L∞=30/255, GPT-4, Bing Copilot and Claude 3 Opus describe a turtle as cannon, with Gemini Advanced as an exception (Table 3); the Stephen Hawking image attack takes effect on GPT-4 from 30/255, and Gemini refuses throughout (Table 4).

   Other ablations and analyses

   • CrossMax step ablation (Table 5, CIFAR-100 self-ensemble, lightweight adversarial training): AB-order topk2 aggregation adv acc 57.81%, mean aggregation 39.06%–48.44%.
   • Learning rate: optimal near about 3.3×10⁻⁵; large learning rates give good clean accuracy but poor robustness (Figure 8a).
   • Training epochs: robust accuracy peaks early then declines (Figure 8b).
   • Spectrum: the power spectrum of multi-resolution attacks is close to natural images (slope about −2), distinct from noise-like attacks (Figure 13).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list four future directions; experiments cover only CIFAR-10/100, and the main results use 128–512 images.

   Limitations and future directions noted by the authors

   • Scaling to larger datasets and models: Future Work states that scalability on larger datasets such as ImageNet and newer architectures needs to be studied (the authors say they expect little problem).
   • Theoretical basis: Future Work states that a deeper understanding is needed of why multi-resolution input and CrossMax bring robustness.
   • Other attack types: Future Work states that this paper only focuses on L∞ perturbations, and that evaluating L2, L1, or semantic adversarial examples would be valuable (the authors expect it to be effective but did not test it).
   • Combining with other defenses: Future Work states that combinations with mechanisms such as adversarial training and certified defenses could be explored.
   • Hyperparameter search: Section 3.1 states that hyperparameter search was very limited, that the resolution combinations etc. were chosen arbitrarily, and the authors believe there is room for improvement.

   Scope of experiments

   • The models tested are ResNet18, ResNet152, and ViT-B/16 (self-ensemble figure), 3 backbone types in total, and the datasets are only CIFAR-10 and CIFAR-100.
   • The main results use 128 test images, CIFAR-100 self-ensemble uses 512, and ResNet18 trained from scratch uses 1024; the paper does not report the number of repetitions.
   • Attack evaluation is RobustBench AutoAttack with L∞=8/255 (the rand version uses only 128 images), and other norm attacks are not covered.
   • Transfer attacks cover GPT-4, Bing Copilot, Claude 3 Opus, Gemini Advanced, and Google Lens, with a single image and a single question per model per attack strength (Table 3, 4).
   • Interpretability conclusions are based on illustrative examples and 8 GPT-4/Gemini queries, and the paper does not report a systematic human evaluation.

   Collapse this answer

6. 06Summarize the main content of the paper

   An adversarial defense with multi-resolution input plus CrossMax self-ensemble; the authors claim that without adversarial training it exceeds SOTA on CIFAR-100 by about 5%.

   A computer vision adversarial robustness paper that proposes using biomimetic multi-resolution input and robust ensembling to give classifiers adversarial robustness by default.

   • Problem: Adversarial examples expose inconsistencies between human and machine vision functions; the authors believe past improvements relied too heavily on adversarial training and rarely rethought the training paradigm itself.
   • Method: Stack the image into a 12-channel input at 4 resolutions with jitter and noise; use the Vickrey-auction-inspired CrossMax (median/top-k after two max normalizations) to aggregate intermediate-layer linear probe predictions into a self-ensemble; the basis is what the authors call "adversarial inter-layer de-correlation"—attack effects are localized by early/middle/late layer.
   • Main results: Under RobustBench AutoAttack (rand, L∞=8/255), without adversarial training the adversarial accuracy is 71.88% on CIFAR-10 and 48.16% on CIFAR-100 (Table 1); the authors state these correspond to 3rd place on the leaderboard and about +5% over the best, respectively.
   • With adversarial training: After lightweight FGSM adversarial fine-tuning, CIFAR-10 reaches 78.13% and CIFAR-100 reaches 51.28% (Table 2); the authors state this exceeds SOTA by about 5% and 9%, with larger gains on the harder CIFAR-100.
   • By-products: The authors state that gradient attacks on this model yield human-understandable semantic changes; the same multi-resolution prior can turn the classifier and OpenCLIP into a controllable image generator, and construct attacks that transfer to commercial vision-language models such as GPT-4 (Table 3, 4).
   • Authors' conclusion: The results support their Interpretability-Robustness Hypothesis; future work needs to verify ImageNet-scale scaling, other norm attacks, and the theoretical basis.

   Collapse this answer

Related papers