Robust physical-world attacks on deep learning models
Robust Physical-World Attacks on Deep Learning Models
The authors propose the RP2 algorithm to generate physical-world adversarial perturbations, attacking real stop signs with black-and-white stickers and achieving targeted attack success rates of 84.8% (LISA-CNN) and 87.5% (GTSRB-CNN) on driving video frames.
White-box attack: the attacker has the network structure and weights of the target classifier (the authors believe this can be approximately obtained through model extraction attacks), cannot control the vehicle system, but can physically modify objects such as road signs; the attack is a targeted attack, and the victim systems are DNN-based road sign classifiers and Inception-v3.
- DNN classifiers are easily misled by adversarial examples, while digital-domain perturbations easily fail in the physical world under changes in distance, angle, lighting, and printing error; safety-critical scenarios such as road sign recognition lack methods for generating and evaluating robust adversarial perturbations under real physical conditions.
- The authors propose RP2: in the optimization objective, take the expectation over sampling both physical and synthetic transformations simultaneously, add a mask to restrict the perturbation to the object surface and disguise it as graffiti, and add an NPS term to model printing color deviation; the evaluation uses a two-stage method of stationary laboratory tests plus driving video tests.
- On the self-built LISA-CNN and GTSRB-CNN, the poster attack achieves 100% targeted success rate in both stationary and driving tests; the sticker attack on GTSRB-CNN achieves 80% stationary and 87.5% on driving frames; on Inception-v3, the microwave and coffee mug sticker attacks achieve targeted success rates of 90% and 71.4% respectively.
- The authors state that the experiments are a white-box setting and that black-box attacks are an open problem; images were manually cropped during evaluation, and under pseudo-random cropping the targeted success rate of LISA-CNN drops to 70% and GTSRB-CNN to 60%; the experiments did not explicitly control ambient lighting.
- Evaluated models
- LISA-CNNGTSRB-CNNInception-v3
- Benchmarks
- LISAGTSRB
- Metrics
- targeted-attack success rateuntargeted attack success rateclassification confidence
Researchers from the University of Washington, UC Berkeley and other institutions propose RP2, a physical adversarial attack algorithm that generates perturbations in the form of printed posters or stickers so that they remain effective under changes in shooting distance and angle, and validate it on a road sign classification task. In the white-box setting, after a black-and-white sticker disguised as graffiti-style art is placed on a real Stop sign, the targeted attack success rate against LISA-CNN is 100% in laboratory tests and 84.8% in video frames from in-vehicle road tests, causing the Stop sign to be recognized as a speed limit 45 sign; the sticker attack against GTSRB-CNN has a success rate of 80% in stationary tests and 87.5% in road tests. The paper also proposes a two-stage evaluation method combining laboratory and road testing, and shows that a single sticker can make Inception-v3 misrecognize a microwave oven as a mobile phone (targeted success rate 90%), suggesting that future defenses cannot rely on physical noise as a protective means.
Why it was selectedThe paper uses laboratory and in-vehicle road tests to show that graffiti-style sticker perturbations can still stably deceive road sign classifiers under changes in distance and angle, and is a foundational reference for physical adversarial attacks.
Paper analysis
What problem does this paper attempt to solve?
How to generate robust physical adversarial perturbations that can still mislead DNN classifiers under changes in physical conditions such as distance and angle.
Core problem: digital-domain adversarial examples easily fail in the physical world under environmental changes (distance, angle, lighting, printing error); how to generate adversarial perturbations that still reliably mislead DNN classifiers under real physical conditions.
- Scenario and motivation: DNNs are being used in the control loops of safety-critical systems such as cars, drones, and robots; the authors take road sign classification as the target domain because road signs are visually simple, the environment is noisy, and it relates to traffic safety.
- Shortcomings of existing methods: the authors say perturbations generated by digital-domain algorithms are too small in magnitude for camera sensors to perceive; perturbations spread across the background while the attacker cannot physically modify the background; and fabrication processes such as printing have color reproduction error.
- Core observation: the authors believe physical perturbations must remain robust to continuous changes in camera distance and angle, because vehicles continuously capture frames from different viewpoints and may vote before deciding.
- Contributions of the paper: propose the attack algorithm RP2 (Robust Physical Perturbations), and propose a two-stage evaluation method of stationary laboratory tests plus driving tests; use black-and-white sticker attacks on real stop signs.
- Threat model:
- Goal: targeted attack, making the classifier classify a modified object (e.g., a stop sign) as an attacker-specified class (e.g., Speed Limit 45).
- Knowledge: white-box, the attacker has the network structure and weights of the target classifier; the authors believe the attacker can obtain an approximate model through model extraction attacks.
- Capability: cannot control the vehicle system, but can physically modify objects at low cost (print posters, apply stickers).
- Victim systems: DNN-based road sign classifiers (LISA-CNN, GTSRB-CNN) and Inception-v3.
What related research exists?
Related work is divided into two categories: digital-domain adversarial examples and physical-domain adversarial examples; this paper introduces sampling of real physical conditions into physical-domain work.
Digital-domain adversarial examples
- Goodfellow et al. (2014): proposed the fast gradient method, constructing adversarial examples using a first-order approximation of the loss function.
- Carlini & Wagner (2017), Liu et al. (2016): proposed optimization-based targeted attack methods. The paper states that these works advanced understanding of digital adversarial examples, whereas this paper examines physical perturbations on real objects under changing environmental conditions.
- Papernot et al. (2016): studied the black-box transferability of digital adversarial examples; the authors accordingly consider that physical black-box attacks may also be feasible.
Physical-domain adversarial examples
- Kurakin et al. (2016): printed adversarial images were still misclassified after being photographed with a phone camera; Athalye & Sutskever (2017): generated adversarial examples robust to 2D synthetic transformations. The paper points out that these works do not modify physical objects, but only print digital perturbation images; this paper's object-constrained poster attack reproduces such attacks and additionally restricts the perturbation to the sign surface.
- Athalye et al. (2017, parallel work): 3D-printed small-scale replicas of perturbed objects. The paper lists three differences: this paper samples both physical and synthetic transformations during optimization, modifies existing objects at real size, and simulates test conditions close to practical use.
- Sharif et al. (2016): printed perturbations on eyeglass frames to attack face recognition, succeeding under stable conditions with small variations in pose, distance and lighting; the authors consider that environmental conditions may vary greatly, so they chose road sign classification, an unconstrained environment.
- Lu et al. (2017): conducted physical adversarial experiments on road sign detectors, claiming that existing detectors cannot be attacked; this paper focuses on classifiers, and attacking detectors is out of scope.
Baseline and benchmark: the baseline is the classification result of unperturbed road signs (100% classified as the true class in stationary tests); the datasets are LISA (taking the 17 most common signs) and GTSRB (German stop signs in the training, validation and test sets replaced with LISA's U.S. stop signs). The authors position this paper as: understanding adversarial examples under the condition that the object itself is physically modified, providing a basis for building robust vision models.
How does the paper address this problem?
In optimization, RP2 samples physical and synthetic transformations, uses a mask to constrain the perturbation region and disguise it as graffiti, and then uses an NPS term to model printing error.
Overall idea: extend single-image adversarial perturbation optimization to optimization that takes the expectation over a distribution of physical conditions, and use a mask to restrict the perturbation to the object surface, disguise it as graffiti, and finally print it as a poster or sticker to attach to a real object. The method is named RP2 (Robust Physical Perturbations).
Analysis of physical-world challenges
- Environmental conditions: the distance and angle of the camera relative to the road sign vary continuously, along with lighting, weather, and stains on the lens or sign.
- Spatial constraints: the attacker cannot modify the background, and the background changes with viewpoint.
- Physical limit of imperceptibility: perturbations that are too small cannot be perceived by the camera, depending on the sensing hardware.
- Fabrication error: printing devices have color reproduction error.
Optimization objective
- Starting from single-image optimization, solve in Lagrangian relaxation form: argminδ λ |δ|p + J(fθ(x+δ), y∗), where J is the loss function, y* is the target class, and λ controls perturbation regularization.
- Model the image distribution X^V, sampling instances xi simultaneously from experimental data collected under real physical conditions and from synthetic transformations (random cropping, brightness changes, spatial transformations); the authors believe that using only synthetic transformations may miss physical effects.
- The final objective is argminδ λ |Mx · δ|p + NPS + Exi ∼ XV J(fθ(xi + Ti(Mx · δ)), y∗), where Mx is the perturbation mask and Ti maps transformations on the object to the perturbation.
Mask and graffiti disguise
- Role of the mask: project the perturbation onto the object surface region, provide spatial locality, and make the perturbation visible but inconspicuous; the attacker can shape the mask into a graffiti style, which the authors say hides the perturbation within human psychological expectations.
- Choice of mask position: the authors say experiments found that mask position affects attack effectiveness, and hypothesize that objects have strong and weak feature regions for classification; first use L1 regularization plus a full-surface mask to obtain a sparse perturbation to locate vulnerable regions, then use L2 to recompute the perturbation on these regions.
Printing error modeling
- Add an NPS (Non-Printability Score) term (from Sharif et al.) to the objective function, measuring the gap between colors in the perturbation and the set of printable colors, to model printer color reproduction error.
Attack implementation and evaluation method
- Two attack forms: object-constrained poster printing attack (perturbation covers the entire sign surface, excluding background) and sticker attack (black-and-white stickers, disguised as graffiti or artistic patterns).
- Optimization settings: ADAM optimizer, β1 = 0.9, β2 = 0.999, ε = 10^-8, η ∈ [10^-4, 10^0].
- Two-stage evaluation: stationary laboratory tests take photos under multiple distance/angle configurations; driving tests use a phone camera in a car, starting recording from about 250 ft away, at vehicle speeds of 0–20 mph, sampling every 10th frame (k = 10). Attack success requires that the clean image at the same distance/angle be correctly classified, to rule out other factors.
What experiments did the paper conduct?
Poster and sticker attacks on LISA-CNN, GTSRB-CNN, and Inception-v3 achieve targeted success rates of up to 100% in both stationary and driving tests.
Experimental setup
- Models tested: LISA-CNN (three convolutional layers plus one fully connected layer, 91% accuracy on the LISA test set), GTSRB-CNN (multi-scale CNN, 95.7% accuracy on the GTSRB test set, 99.4% accuracy on 181 stop sign images collected by the authors), Inception-v3 (pretrained).
- Datasets: LISA (17 most common signs); GTSRB (German stop signs replaced with LISA's U.S. stop sign images).
- Attack types: object-constrained poster attack (Stop→Speed Limit 45, subtle attack on Right Turn sign), sticker attacks (disguised as graffiti, disguised as artistic patterns; GTSRB-CNN target is Speed Limit 80; Inception-v3 targets are phone, cash machine).
- Metrics: targeted attack success rate (requires clean images under the same configuration to be correctly classified), non-targeted success rate, target class confidence.
- Evaluation protocol: stationary tests cover distances of 5'–40' and angles of 0°–60° (Inception-v3: 2'–20' and 8''–32''); driving tests sample every k = 10 frames, with baselines of two clean sign videos; also tried k = 15, and the authors say the success rate showed no obvious change.
- Baseline: unperturbed road signs, classified as the true class 100% of the time in both stationary tests and driving baselines.
Main results
According to Table 1, Table 3, and Sections 4.3–4.5, the targeted success rates of each attack:
Swipe horizontally to read the table
Target model Attack Stationary test Driving test LISA-CNN Subtle poster (Stop) 100% 100% LISA-CNN Subtle poster (Right Turn) 73.33% not reported LISA-CNN Graffiti-disguised sticker 66.67% not reported LISA-CNN Art-disguised sticker 100% 84.8% GTSRB-CNN Art-disguised sticker 80% 87.5% Inception-v3 Microwave sticker 90% not reported Inception-v3 Coffee mug sticker 71.4% not reported - Poster attack: the authors report that the Stop sign was classified as Speed Limit 45 under all stationary test configurations, with an average target class confidence of 80.51% (Table 2).
- Right Turn exception: 4 of 15 distance/angle configurations were not classified into the target class, but were still misclassified as Yield or Added Lane; the authors hypothesize that warning signs look similar, so a small perturbation suffices to confuse.
- Graffiti sticker weaker: the authors say region mismatches may cause the lower success rate of the LOVE-HATE graffiti sticker.
Effect of image cropping
- What was tested: in the main experiments, each image was manually cropped before classification; a supplementary experiment evaluated the art-disguised attack using pseudo-random cropping that ensures most of the sign is within the frame.
- Results: LISA-CNN average targeted success rate 70%, non-targeted 90%; GTSRB-CNN average targeted 60%, non-targeted 100%.
- Authors' interpretation: cropping has some effect on targeted success rate; the authors mention that their subsequent work attacked detectors that require no cropping using an improved version of RP2.
Other ablations and analyses
- Frame sampling interval: comparing k = 15 with k = 10, the authors say the attack success rate shows no obvious change.
- Coffee mug non-targeted success rate: 100% (targeted 71.4%, Table 7).
- Lighting: the experiments did not explicitly control ambient light; the experimental data cover indoor to outdoor lighting.
Negative results and exceptions
- The Right Turn subtle poster attack has a targeted success rate of 73.33%, lower than other attacks (Table 4).
- The graffiti-disguised sticker has a stationary success rate of 66.67%, the lowest among road sign attacks (Table 1).
- The Inception-v3 coffee mug attack was classified as non-target classes such as Pitcher under configurations including 24'' 15◦, 28'' 0◦, 32'' 0◦, 32'' 15◦ (Table 7).
What are some points that can be further explored?
The authors state that black-box attacks are an open problem; the experiments are white-box, use manual cropping, do not control lighting, and cover only three classifiers.
Limitations and future directions noted by the authors
- Black-box setting not validated (Section 5 Discussion): the authors state that evaluating RP2 in a black-box setting is an open problem.
- Effect of frame sampling choice unknown (Section 4.2): the authors state that an autonomous vehicle may classify every j frames and vote, and whether the frame choice j affects attack accuracy is an open problem (in experiments j = 10, and no obvious change was seen at j = 15).
- Lighting not controlled (Section 4.2): the authors state that the current experiments do not explicitly control ambient lighting.
- Detector attacks out of scope (Section 2): the authors state that attacking detectors is out of the scope of this paper, and mention that their recent work has extended RP2 to attack YOLO detectors.
Scope of experimental coverage
- The models tested are LISA-CNN, GTSRB-CNN, and Inception-v3, 3 in total, all classifiers with what the authors call standard architectures.
- The attack knowledge setting is white-box; the paper does not conduct experiments in a black-box setting.
- Physical objects cover stop signs, Right Turn warning signs, microwaves, and coffee mugs; driving tests are conducted only on stop signs, at vehicle speeds of 0–20 mph, on straight roads without curves.
- Images are manually cropped before classification; pseudo-random cropping is a supplementary experiment, used only for the art-disguised attack.
- The paper does not report the number of optimization iterations per attack, or the time or cost required to generate perturbations.
Summarize the main content of the paper
The poster and sticker perturbations generated by RP2 reliably mislead road sign classifiers across multiple distances and angles, with a targeted success rate of up to 100% on driving video frames.
A CVPR 2018 physical adversarial example attack paper that studies how to make printed perturbations still mislead DNN visual classifiers under changes in real physical conditions.
- Problem: Digital-domain perturbations easily fail under changes in distance, angle, lighting, and printing errors, while scenarios such as road sign recognition are safety-critical.
- Method: RP2 takes the expectation over the distribution of physically collected data and synthetic transformations during optimization, uses a mask to restrict perturbations to the object surface and disguise them as graffiti, and adds an NPS term to model printing color differences; the attack forms are object-restricted posters and black-and-white stickers; evaluation uses a two-stage method of stationary laboratory tests plus driving tests, and the threat model is white-box.
- Main results:
- On LISA-CNN, the poster attack achieves 100% targeted success rate in both stationary and driving tests, with an average confidence of 80.51% for the target class (Table 2, 3).
- The LISA-CNN disguised art sticker achieves 84.8% success rate on driving frames; the GTSRB-CNN sticker attack achieves 80% stationary and 87.5% on driving frames (Table 1, 5).
- On Inception-v3, the microwave and coffee mug sticker attacks achieve targeted success rates of 90% and 71.4% respectively (Table 6, 7).
- Exceptions: the Right Turn subtle poster attack is 73.33%, and the disguised graffiti sticker is 66.67%; under pseudo-random cropping, the targeted success rates of LISA-CNN and GTSRB-CNN drop to 70% and 60%.
- Authors' conclusion: Adversarial examples that are robust to changes in distance/angle exist in the physical world, and future defenses should not rely on physical noise as protection; the authors state that black-box attacks and detector attacks are left for future work.