Adversarial examples that simultaneously deceive computer vision models and time-limited humans

1. 01What problem does this paper attempt to solve?

   Whether transferable adversarial examples generated for computer vision models can also affect human observers under time-limited viewing.

   Core question: whether transferable adversarial examples constructed for computer vision models can transfer to the human visual system and affect human classification judgments.

   • Problem setting and importance: adversarial examples cause models to misclassify by applying small perturbations to images (e.g., recognizing a school bus as an ostrich), but the authors point out that whether humans exhibit similar errors remains an open question; the prior understanding was that such transferable adversarial examples have no effect on human visual perception, but there had been no thorough empirical test.
   • Shortcomings of existing methods: most adversarial example construction algorithms require access to the model's architecture and parameters for gradient-based optimization, whereas the human brain cannot provide similar access, so these methods appear inapplicable to constructing adversarial examples for humans.
   • Core observation and hypothesis: the authors note three clues—adversarial examples can transfer across models, training sets, and algorithms, and even from the digital world to the physical world; stronger adversarial examples optimized against multiple models or viewpoints sometimes exhibit human-interpretable semantic features (Figure 1b, c); and deep representations of CNNs are similar to primate visual pathway activity. The authors therefore hypothesize that, with appropriate design, adversarial examples can transfer from models to humans.
   • What the paper proposes: a set of methods combining machine learning, neuroscience, and psychophysics—using black box transfer attack techniques, a retinal layer simulating the initial visual processing of the human eye, and time-limited psychophysical experiments—to construct and test adversarial examples that deceive both computer vision models and time-limited humans.
   • Threat model:
     • Attacker goal: generate targeted adversarial examples that cause the classifier to misclassify an image of class A as a specified class B (constructed bidirectionally).
     • Knowledge: white box for the 10 models used for generation (architectures and parameters known); black box for the test models and humans (no access to architecture or parameters).
     • Capability: apply a per-image perturbation with l∞ norm equal to a fixed ε to each image (adv condition ε=32, false condition ε=40).
     • Victim systems: ImageNet CNN classifiers (including test models that did not participate in generation) and time-limited human observers.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: adversarial examples and their transferability, similarities between biological and artificial vision, and differences between the two.

   Adversarial examples and transferability

   • Szegedy et al. [39], Goodfellow et al. [13]: proposed and defined the adversarial example phenomenon; Goodfellow et al. [12] defined adversarial examples as "inputs intentionally designed by an attacker to make the model err." Figure 1a reproduces the classic example from [13] of a panda misclassified as a gibbon.
   • Papernot et al. [30, 31], Liu et al. [26]: studied the transferability of adversarial examples across models, training sets, and algorithms, and black box attacks; Liu et al. [26] pointed out that optimizing against multiple models can improve transferability to any sixth model, and this paper's ensemble attack builds on that work.
   • Kurakin et al. [24]: found that adversarial examples can transfer from the digital world to the physical world; this paper's iterative targeted attack method is also taken from that work.
   • Athalye and Sutskever [2], Brown et al. [4]: constructed adversarial examples robust to geometric transformations or multiple viewpoints; the authors point out that these examples exhibit human-interpretable semantic features (Figure 1b, c), which is a clue that "machine-to-human transfer may be feasible."

   Similarities between biological and artificial vision

   • Cadieu et al. [6], Yamins and DiCarlo [43]: found that deep-layer CNN activity can predict primate visual pathway activity; the cortical object recognition model of Riesenhuber and Poggio [36] is similar to modern CNNs in many respects.
   • Kümmerer et al. [21, 22], Gatys et al. [10], Freeman and Simoncelli [9]: CNNs can predict human fixation points, mid-level representations capture styles meaningful to humans, and CNN-like models can construct psychophysical metamers.
   • Geirhos et al. [11], Rajalingham et al. [35]: compared error patterns of humans and neural network classifiers in psychophysical experiments.

   Differences between biological and artificial vision

   • Van Essen and Anderson [41]: the spatial resolution of the primate eye varies with eccentricity, with high resolution within about 5° of the fovea and then a linear decline with eccentricity; this paper's retinal layer is based on their (macaque visual system) measurement data.
   • Eckstein et al. [8]: experiments found that humans miss large objects in scenes whereas deep networks do not; the authors use this to illustrate that the two have qualitative differences in error patterns.
   • Olshausen [29], Ibbotson and Krekelberg [18]: visual cortex has far more feedback than feedforward connections, exhibits recurrent dynamics, and the human brain actively explores scenes via saccades; the authors state that these differences are mitigated by time-limited presentation.

   Baselines and controls: this paper has no comparison baseline methods; the experimental controls are the paper's own image (clean image) and flip (vertically flipped perturbation) conditions, and the dataset is ImageNet [7].

   The authors position this paper as combining three elements: black box transfer attacks, model modification to match human initial visual processing, and time-limited psychophysical evaluation, providing the first empirical test of the effect of transferable adversarial examples on humans (the paper does not use the wording "first," only stating that previously there was "no thorough empirical study").

   Collapse this answer

3. 03How does the paper address this problem?

   Iterative targeted attacks were performed on a 10-model ensemble with a retinal preprocessing layer to generate adversarial images, which were then tested on humans using time-limited two-alternative forced choice psychophysical experiments.

   Overall approach: using the idea of a black box transfer attack, generate strongly transferable targeted adversarial examples on an ensemble of CNNs that simulate the initial processing of the human eye, then test their effect on human classification through time-limited psychophysical experiments.

   Data and class system

   • Image source: ImageNet [7]; because its 1000 fine-grained classes are too specialized, the authors merged them into 6 coarse classes familiar to participants: dog, cat, broccoli, cabbage, spider, snake.
   • Three experimental groups: Pets (dog/cat), Hazard (spider/snake), Vegetables (broccoli/cabbage); each session uses only one group, with binary classification within the group.
   • Pre-screening: images whose classes were clearly distinguishable by overall color or brightness were excluded (e.g., in the pets group, images with large green lawns were almost all dogs), to prevent participants from using such strategies.

   Model ensemble and retinal layer

   • Ensemble composition: k=10 CNNs trained on ImageNet, with architectures being instances of Inception V3, Inception V4, Inception ResNet V2, and ResNet V2 50/101/152 (some with public pretrained weights, some self-trained; see Appendix E).
   • Retinal layer: each model is preceded by a differentiable retinal preprocessing layer that applies position-dependent blur to the image according to eccentricity, approximating the input the human retina's lattice transmits to the visual cortex; the degree of blur is computed based on eccentricity-resolution measurements of the macaque visual system [41] and the viewer-screen geometry (Appendix B).
   • Design rationale: the authors state that this layer restricts the CNN's input to the information equally available to the human visual system, making it more likely that adversarial examples transfer from models to humans.

   Adversarial example generation

   • Optimization objective: given classifier k, target class ytarget, and maximum perturbation ε, find X_adv minimizing −log(P(y_target|X_adv)), subject to ||X_adv−X||_∞=ϵ; coarse-class probabilities are obtained by summing the probabilities of the fine classes within the class (Appendix C).
   • Attack algorithm: iterative targeted attack [24], step size α=2, gradient descent on the ensemble loss; the ensemble is the geometric mean of the coarse-class probabilities of each model (Appendix D).
   • Selection criteria: only examples that successfully attack all 10 models are retained (adv condition); the false condition requires at least 7/10 models to succeed, to increase the transfer rate.

   Human psychophysical experiments

   • Participants and apparatus: 38 participants with normal or corrected-to-normal vision; seated 61 cm from a high-refresh-rate screen (ViewSonic XG2530), using a response-time box for two-alternative forced choice key presses, with the class-key mapping randomized per session.
   • Trial procedure: fixation point 500–1000 ms → image (15.24 cm square, 14.2° visual angle) presented for 63 ms (71 ms in some sessions) → 10 high-contrast binary random masks each 20 ms → response within 2200 ms (2500 ms in some) after the masks end; each participant completed 140–950 trials.
   • Four experimental conditions: image (clean image, rescaled to [40, 255−40]); adv (with ε=32 targeted perturbation, which the authors state is perceptible to humans at this magnitude but preserves the class); flip (perturbation vertically flipped before being superimposed, with statistical properties almost identical to adv, used to rule out the explanation that "the perturbation merely degrades image quality"); false (the true class is not among the two options, forcing participants to err, testing whether the perturbation can control their choice between the two wrong options, ε=40).
   • Success criteria: in the false condition, whether the probability of participants choosing the target class is above the 0.5 chance level; in the adv condition, whether accuracy is lower than the image and flip controls; statistical tests are two-tailed t-tests, Pearson Chi-2 GLM, one-way ANOVA, etc.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Adversarial examples had an attack success rate of 57%–89% against the test models and affected timed human choices and accuracy (pets, vegetables groups p < 0.001).

   Experimental setup

   • Models tested: the generation ensemble is 10 CNNs (instances of Inception V3/V4, Inception ResNet V2, ResNet V2 50/101/152, some preceded by a retinal layer and trained with [40, 255−40] rescaling data augmentation); the two models tested for transfer are an adversarially trained Inception V3 [23] and ResNet V2 50, neither of which participated in generation.
   • Dataset: ImageNet training set images, merged into 6 coarse classes and 3 experimental groups (Pets, Hazard, Vegetables); Appendix F lists all images used.
   • Human experiments: 38 participants, each with 140–950 trials; images presented for 63 ms (71 ms in some), followed by 10 masks of 20 ms each.
   • Metrics: on the model side, accuracy and attack success; on the human side, probability of choosing the target class (false condition), accuracy (image/adv/flip conditions), and reaction time.
   • Statistical methods: two-tailed t-tests (against the 0.5 chance level or between groups), Pearson Chi-2 GLM, one-way ANOVA, Pearson correlation; error bars are ±SE.
   • Control conditions: flip (vertically flipped perturbation) is used to control for the explanation of image quality degradation; the false condition is used to test the perturbation's control over the direction of choice at a fixed error rate.

   Main results

   Model-side results (according to Table Supp.4 and Supp.6; triplets are the pets, hazard, vegetables groups):

   Swipe horizontally to read the table

   Model	adv accuracy (%)	flip accuracy (%)	adv attack success (%)	flip attack success (%)
   ResNet V2 50 (test model)	8.7, 9.4, 13	93, 91, 85	87, 85, 57	1.3, 0.0, 0.0
   Inception V3∗∗ (test model, adversarially trained)	6.0, 6.9, 17	95, 92, 94	89, 87, 74	1.5, 0.5, 0.0

   • The 10 models within the generation ensemble all had 0.0% accuracy and 100% attack success in the adv condition (Table Supp.3, Supp.5), which is the expected result of the generation selection; both models in the table above did not participate in generation.
   • Main text summary: the attack success rate of the adv and false conditions against the test models ranges from 57%–89% (varying with class and condition), and the proportion of flip condition changing the test models' predictions is below 1.5% in all conditions; the authors use this to validate the reasonableness of flip as a control (Section 4.1).

   Human-side results (Figure 3, Table Supp.1):

   • false condition: in all three experimental groups, the probability of participants choosing the adversarial target class was above the 0.5 chance level (Figure 3a, two-tailed t-test); effect strength was strongest in the hazard group, second in pets, and weakest in vegetables, with between-group differences p < 0.05 (Pearson Chi-2 GLM).
   • Lower accuracy in the adv condition: for most participants, average adv accuracy was lower than image (pets 29/35, hazard 19/24, vegetables 21/32, Table Supp.1); the number of participants with adv lower than flip was pets 22/35, hazard 16/24, vegetables 23/32. Average accuracy for adv lower than flip was p < 0.001 in the pets and vegetables groups and p = 0.05 in the hazard group (Figure 3b).
   • Authors' interpretation: the authors state that these results indicate that the combination of perturbation direction and specific image is perceptually related to the features the human visual system uses for classification, constituting evidence that strong black box adversarial attacks can transfer from CNNs to humans.

   Reaction time analysis

   • false condition reaction times: between-group differences in mean reaction time p < 0.001 (one-way ANOVA, Figure 4a); the reaction time pattern was negatively correlated with the perceptual bias pattern (Pearson correlation = −1, p < 0.01), i.e., the hazard group decided fastest and vegetables slowest.
   • Longer reaction times in the adv condition: in all three groups, mean reaction time in the adv condition was longer than the control, but only two comparisons reached statistical significance (Figure 4b).
   • Binning by reaction time: in the false condition, for fast responses (0–33rd percentile) the bias toward choosing the target class was significant relative to 0.5, while for slow responses it was not significant (Figure 4c). The authors propose an explanation: in the false condition, transfer is accompanied by higher confidence, whereas in the adv condition adversarial features compete with true-class features and are accompanied by lower confidence.

   Other ablations and analyses

   • Perturbation magnitude ε (Figure Supp.1b): the authors state that a perturbation of ε=8 already makes the example image perceptually closer to the target class, and consider that transfer may also hold for smaller ε; this is an intuitive demonstration on a single example image, not a human experiment.
   • Retinal layer (Figure Supp.1c): after removing the retinal layer, the similarity of the example image to the target class decreased; the authors therefore consider matching initial visual processing to be an important factor for transfer.
   • Ensemble size (Figure Supp.1d): generated with 1–10 models at ε=32; the authors state that as the number of models decreases, the similarity of the perturbation to the target class decreases, supporting the importance of the ensemble.
   • Long-presentation control (Figure 3c): one spider adversarial image, when briefly presented, led 6 of 13 participants to report snake (67%), whereas under long presentation (post-experiment email survey, 13 people) only 4 did (25%); the authors state that the perturbation preserves the class for humans without time limits.
   • Mechanism observations (Figure 5, Section 5.2): the authors non-controlledly observed that the adversarial perturbation appears to act by disrupting object edges (especially mid-frequency modulation perpendicular to edges), enhancing edges, modifying texture, and exploiting dark regions (small ε perturbations have greater perceptual magnitude in dark regions); the authors explicitly state that no controlled experiment was designed to prove any specific mechanism.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list open questions including mechanism validation, design sensitivity, and deception without time limits; the human experiment scale was 38 participants and 6 coarse classes.

   Limitations and future directions noted by the authors

   • Mechanism not proven (Section 5.2): the authors explicitly write that "no controlled experiment was designed to prove that the adversarial examples act in any specific way"; the patterns in Figure 5 such as edge disruption and texture modification are only informal observations.
   • The ground truth problem for humans without time limits (Section 5.1): the authors state that to deceive humans without time limits in the future, the difficult problem of "obtaining a better ground truth signal than human visual annotation" must be solved.
   • Sensitivity of transfer to design elements unknown (Section 5.4): the authors raise open questions—how transfer depends on ε, whether the model ensemble is critical, and whether the retinal preprocessing layer can be removed; the authors conjecture that both the retinal layer and the ensemble are important and that ε could be smaller, and Figure Supp.1 is only a preliminary exploration.
   • Two explanations for the robustness of humans without time limits not distinguished (Section 5.3): the authors state that one possibility is that humans without time limits are inherently more robust by virtue of top-down or lateral connections, and another is that humans without time limits remain vulnerable and the adversarial examples simply cannot transfer from feedforward networks due to architectural differences; the results of this paper cannot distinguish the two.

   Scope of experimental coverage

   • The models tested are 10 generation ensemble instances of 6 architectures plus 2 test models (ResNet V2 50, adversarially trained Inception V3), all ImageNet feedforward CNNs (Section 3.1.2, Table Supp.2).
   • The human experiments are 38 participants, 6 coarse classes, 3 experimental groups, and a two-alternative forced choice task, with 140–950 trials per person (Section 3.2).
   • The perturbation setting is an l∞ constraint, with ε=32 in the adv condition and ε=40 in the false condition, and step size α=2 (Section 3.2.2, Appendix D).
   • Human effects were measured only under brief presentation of 63 ms (71 ms in some) plus masks; the long-presentation condition is only one image reported in Figure 3c and an email survey of 13 people.
   • The paper does not report specific numerical values for the probability of choosing the target class in the false condition (only showing graphically that it is above 0.5), nor does it report the number of replications of the human experiments.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors constructed adversarial examples that simultaneously deceive multiple CNNs and time-limited humans; the authors state that this constitutes a class of illusion shared by the two types of systems.

   An empirical study on cross-species transfer of adversarial examples: whether transferable adversarial examples generated for computer vision models can affect human observers under time-limited viewing.

   • Problem: adversarial examples typically require white box access to the model and appear impossible to construct against the human brain; the prior understanding was that such transferable adversarial examples do not affect human perception, but there was a lack of empirical testing.
   • Method highlights: on an ensemble of 10 ImageNet CNNs (Inception V3/V4, Inception ResNet V2, ResNet V2 50/101/152), preceded by a differentiable retinal layer simulating the eccentricity-dependent resolution of the human eye, iterative targeted attacks were used to generate adversarial images with an l∞ constraint (ε=32); 38 participants performed two-alternative forced choice classification under 63 ms presentation plus masks, with four conditions: image, adv, flip (flipped perturbation control), and false (forced error).
   • Main results:
     • The attack success rate of the adversarial examples against the two test models that did not participate in generation was 57%–89%, and the flip control changed predictions by less than 1.5% (Section 4.1, Table Supp.6).
     • In the false condition, the probability of participants in all three experimental groups choosing the adversarial target class was above the 0.5 chance level (Figure 3a).
     • Average accuracy in the adv condition was lower than the flip control, with p < 0.001 in the pets and vegetables groups and p = 0.05 in the hazard group (Figure 3b); for most participants, adv accuracy was lower than image (e.g., 29/35 in the pets group, Table Supp.1).
     • Reaction time was negatively correlated with the attack effect: in the false condition, the group with the stronger bias decided faster (Pearson correlation = −1, p < 0.01), and the bias was significant for fast responses but not for slow responses (Figure 4).
   • Authors' conclusions and implications: the authors state that this constitutes a new class of illusion shared by computer vision models and the human brain; transferable examples deceive time-limited humans but not humans without time limits, suggesting that top-down or lateral connections may be related to human robustness and are worth exploring in machine learning security research; the authors also state that there is a risk of images being manipulated to influence human viewers' responses (e.g., affecting trust perception of photographs of politicians).

   Collapse this answer

Related papers