Skip to content

Generating Semantic Adversarial Examples with Differentiable Rendering to Analyze and Improve Neural Networks

Analyzing and Improving Neural Networks by Generating Semantic Counterexamples through Differentiable Rendering

arXiv· arXiv:1910.00727· Lakshya Jain, Varun Chandrasekaran, Uyeong Jang, Wilson Wu, Andrew Lee, Andy Yan, Steven Chen, Somesh Jha, Sanjit A. SeshiaUniversity of California, Berkeley; University of Wisconsin-MadisonIndexed by this site · 23 days ago· Originally publishedDefenses

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors use differentiable rendering to move adversarial attacks into the semantic parameter space to generate semantic counterexamples, and the authors state that using them for data augmentation can improve the semantic robustness of classification models and transfer across models.

Threat modelThe attacker targets a DNN classifier or detector, perturbing the input within an ε-neighborhood of the semantic feature space Θ (pose, color, lighting, etc.), using gradients of a differentiable renderer R to generate semantic counterexamples that cause misclassification or missed detection; the attack is white-box (requires gradients of the model and renderer), and the victim systems are image classification and object detection networks.

Questions
DNNs are fragile to physically grounded semantic transformations such as rotation, translation, and lighting, whereas adversarial machine learning mostly focuses on pixel-level perturbations; the authors believe there is a lack of effective techniques for generating semantic counterexamples and using them to improve semantic robustness.
Methods
Define a semantic feature space and, with the aid of a differentiable renderer, rewrite i-FGSM, PGD variants, and CW into semantic versions (s prefix), performing gradient perturbation on semantic parameters to generate counterexamples; additionally propose an information worth metric combining classifier confidence and LPIPS realism to measure the value of augmented samples.
Experiments and results
On ShapeNet, sCW counterexamples reduce VGG-16 overall accuracy from 0.986 to 0.485; after retraining with counterexamples, the model achieves over 0.9 accuracy on the test sets of each attack method; counterexamples generated by VGG-16 transfer to ResNet-50; in the detection task, multi-parameter counterexamples reduce SqueezeDet's mAP from 99.4 to 65.9.
Limitations and future work
The authors state that future directions include testing semantically robust models in a CPS control loop and making the algorithm aware of the rest of the control loop; 3D-SDN's poor rendering quality leads to low realism scores, and its counterexample generation is time-consuming, so transfer tests with multiple hyperparameters were not performed.
Experimental setupVGG-16、ResNet-50 et al. · ShapeNet、VKITTI et al. · normalized accuracy、mAP et al.
Evaluated models
VGG-16ResNet-50SqueezeDetYOLOv3
Benchmarks
ShapeNetVKITTIKITTI
Metrics
normalized accuracymAPrecallinformation worthFIDLPIPS
AI summaryThe researchers propose using differentiable rendering to generate semantic adversarial examples, and thereby analyze deep neural networks and improve their robustness through data augmentation. The method uses off-the-shelf adversarial machine learning attacks to change semantic features in images such as rotation, translation, lighting, or camera pose, generating images that can cause misclassification or misdetection. The authors validate it on image classification and object detection networks. In classification tasks, using semantic adversarial examples for data augmentation can improve generalization performance, enhance robustness to semantic transformations, and transfer across models. Compared with sampling-based semantic augmentation, the data generated by this method is more informative and more sample-efficient.

The researchers propose using differentiable rendering to generate semantic adversarial examples, and thereby analyze deep neural networks and improve their robustness through data augmentation. The method uses off-the-shelf adversarial machine learning attacks to change semantic features in images such as rotation, translation, lighting, or camera pose, generating images that can cause misclassification or misdetection. The authors validate it on image classification and object detection networks. In classification tasks, using semantic adversarial examples for data augmentation can improve generalization performance, enhance robustness to semantic transformations, and transfer across models. Compared with sampling-based semantic augmentation, the data generated by this method is more informative and more sample-efficient.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    How to generate physically grounded semantic counterexamples and use them to analyze and improve the semantic robustness of DNNs.

    Core problem: how to efficiently generate semantic counterexamples—inputs where a small, physically grounded change in the semantic feature space causes misclassification or missed detection—and use them to analyze DNNs and improve semantic robustness through data augmentation.

    • Scenario and importance: DNNs are already used in safety-critical CPS such as autonomous driving, but their performance is fragile to input transformations; the authors argue that realistic semantic transformations such as translation, rotation, lighting, and camera pose are more likely to occur in the real world, and models should have semantic robustness.
    • Shortcomings of existing methods: the authors state that most of the adversarial machine learning literature focuses on pixel-level, norm-constrained perturbations, which are usually unrealizable in the real world; while methods that explicitly sample the semantic parameter space are slow, imprecise, and costly (citing [23, 24]).
    • Core observation: the differentiable renderer R maps the semantic feature vector θ to an image x and is differentiable with respect to θ, so off-the-shelf adversarial attacks can be moved from pixel space to semantic space, using the chain rule to compute gradients and generating semantic counterexamples in a sample-efficient way.
    • Contributions: a framework for generating semantic counterexamples (three constructions: sFGSM/si-FGSM, sGD, sCW), an information worth metric for measuring the value of augmented samples, and validation on classification (VGG-16, ResNet-50 + ShapeNet + Redner) and detection (SqueezeDet, YOLOv3 + VKITTI + 3D-SDN).
    • Threat model:
      • Goal: cause the classifier to misclassify or the detector to miss/falsely detect (e.g., missing the gray foreground car in Figure 1).
      • Knowledge: white-box—the attack requires gradients with respect to the renderer R and the model F (chain rule).
      • Capability: perturb θ (pose, color, lighting, mesh, etc.) within the ε-neighborhood of the semantic feature space Θ, generating images through differentiable rendering.
      • Victim systems: image classification networks (VGG-16, ResNet-50) and object detectors (SqueezeDet, YOLOv3).
  2. What related research exists?

    Related work is divided into three groups: differentiable rendering, pixel-level adversarial attacks, and semantic/physical attacks; the authors state that this paper's generality lies in adapting to arbitrary renderable semantic features.

    Differentiable rendering and inverse graphics

    • Loper et al. (OpenDR) and Kato et al. (Neural 3D Mesh Renderer) — proposed differentiable rendering pipelines; Kulkarni et al. combined de-rendering and rendering to learn interpretable image representations.
    • Li et al. (Redner) — a general-purpose differentiable ray tracer that can compute gradients with respect to arbitrary semantic parameters such as camera pose, geometry, materials, and lighting; the classification experiments in this paper use it.
    • Yao et al. (3D-SDN) — obtains semantic, geometric, and texture representations via de-rendering and then re-renders with a generative model; the detection experiments in this paper use it.

    Pixel-level adversarial attacks

    • FGSM (Goodfellow et al.), PGD (Madry et al.), CW (Carlini & Wagner), DeepFool, Papernot et al. — generate norm-constrained pixel-level perturbations; the paper points out that these manipulations are usually unrealizable in the real world. The three semantic attacks in this paper are adapted from i-FGSM, PGD variants, and CW.

    Semantic and physical-world attacks

    • Eykholt et al.'s physical-world attacks, Bhattad et al.'s color/texture manipulation, Geirhos et al.'s discovery of texture bias, Engstrom et al.'s spatial rotation transformations, Sharif et al.'s facial recognition attacks — each targets specific semantic features.
    • Song et al. use a GAN to perform class-conditional search in latent space to generate semantic adversarial examples; the paper points out that they construct samples from scratch rather than perturbing existing data points, and do not use gradient methods, which limits pipeline flexibility.
    • Dreossi et al. (semantic adversarial deep learning, counterexample-guided data augmentation) — explicitly sample the semantic parameter space; the paper points out that such methods are slow and expensive, and this paper uses their Halton sampling as a baseline.

    Closest work: Xiao et al. (MeshAdv, using differentiable rendering to change shape and texture) and Qiu et al. (SemanticAdv, using generative models for attribute-conditional image editing). The authors state that this paper's distinction lies in generality — it applies as long as semantic features can be rendered via a differentiable renderer — and demonstrate counterexamples used for data augmentation and transfer across classification DNNs.

    Baselines and benchmarks: augmentation baselines are random sampling and Halton sampling (each over two pose ranges, [-0.75, +0.75] and [-0.3, +0.3] radians); datasets are ShapeNet (classification) and VKITTI/KITTI (detection).

  3. How does the paper address this problem?

    Move adversarial attacks from pixel space to semantic feature space: perturb semantic parameters using gradients of a differentiable renderer, and use information worth to measure the value of augmented samples.

    Overall idea: define a semantic feature space Θ and a differentiable renderer R, rewrite an off-the-shelf adversarial attack A into a semantic version sA, perform gradient perturbation on θ to generate semantic counterexamples, and then use them for data augmentation. The core of the method is the combined framework of "renderer + attack".

    Problem setting and formalization

    • Semantic feature space Θ: an element θ describes the environment (object pose, color, texture, camera pose, lighting, weather, background, etc.); the rendering process R maps θ to an image x, and the model F maps x to an output y.
    • Semantic robustness: a small change in θ should only cause a small change in the output, i.e., ∀θ′ ≈Θ θ, F(R(θ)) ≈ F(R(θ′)); a θ̃ that violates this property is a semantic counterexample.
    • Generating semantic counterexamples is formalized as the optimization problem minπ ∈ Θ ν(π) s.t. F(R(θ+π)) ≠ F(R(θ)), i.e., finding in the semantic space the smallest perturbation π that causes misclassification; the counterexample is θ̃ = θ + π*.

    Attack construction (Appendix B)

    • General rewriting rule: turn attack A into sA—replace δ with π, replace x + δ with θ + π, and use the chain rule through R to compute the gradient of the loss with respect to θ.
    • si-FGSM: iteratively update θ in the direction of the gradient sign with step size α; the loss uses the raw (unnormalized) score of the correct class rather than the softmax probability (following Simonyan et al.).
    • sGD: a PGD variant that projects back to the ε-neighborhood of θ0 after each update; the projection operator also keeps parameters within the feasible set (e.g., translation does not move the car off the road).
    • sCW: solves min |π|p + c · f(θ+π), where p = 1, c = 0.1, f is the hinge form of the difference between the largest non-target-class logit and the target-class logit, optimized with the Adam optimizer (learning rate η).

    Augmentation quality metrics (§3)

    • information worth: based on Shannon entropy, measures the degree of inconsistency between augmented samples and the current classifier's predictions; inconsistent (and realistic) samples are more useful for retraining correction.
    • Confidence and realism weighting: softmax values are used as the score membership of a point in each subset; realism ρ(x) = 1 − d(x, x*)/dmax, where d is the LPIPS distance; samples with low ρ are downweighted in metric computation.

    Implementation and pipeline (§4.1, Appendices D, E)

    • Classification pipeline: Redner receives vertex, lighting, material, and pose information to render the object, alpha-blends it with the background, and feeds it into VGG-16/ResNet-50; gradients are computed with respect to semantic parameters and perturbed (Figure 3).
    • Detection pipeline: 3D-SDN first de-renders the input image into intermediate representations (semantic, graphic, texture maps), perturbs them, and re-renders (Figure 4).
    • Semantic parameters: VKITTI/3D-SDN use color, weather, foliage, rotate, translate, mesh; ShapeNet/Redner use pose, vertex, lighting (lighting does not affect classification accuracy and is omitted).
    • Hyperparameters: on VGG-16 all three attacks iterate 5 times, with per-parameter step sizes (e.g., si-FGSM's αvertex = 0.002, αpose = 0.15; sGD's εvertex = 0.05, εpose = 1.0; sCW's ηvertex = 0.01, ηpose = 0.30, Table 10); the authors state the hyperparameters were determined by independent visual inspection by 3 viewers.
    • Training: for classification, ImageNet-pretrained models are retrained on the last layer for 20 epochs; robust models are trained for another 20 epochs after randomly replacing half of the training set with counterexamples. For detection, SqueezeDet is trained for 24000 steps on 6339 re-rendered images, and the robust model is trained for 6000 steps after replacing a quarter of the training set.
  4. What experiments did the paper conduct?

    On ShapeNet classification and VKITTI detection, the informativeness, realism, post-augmentation robustness, and cross-model transferability of the counterexamples are validated; sCW reduces VGG-16 accuracy from 0.986 to 0.485.

    Experimental setup

    • Models under test: classification uses VGG-16 and ResNet-50 (the main text focuses on VGG-16; ResNet-50 is in Appendix F); detection uses SqueezeDet (main) and YOLOv3 (transfer test, Appendix G.5).
    • Datasets: ShapeNet (12 classes, 3670 test / 12843 train / 1835 validation images, Table 13; §4.4 uses a smaller test set of 232 images); VKITTI (6339 train / 2082 test images, containing cars and buses); YOLOv3 is trained on KITTI.
    • Attack methods: sCW, si-FGSM, sGD (detection uses only si-FGSM); baselines are random sampling and Halton sampling, each with a large pose range [-0.75, +0.75] and a small range [-0.3, +0.3] radians, sampling 5 samples per benign point.
    • Metrics: per-class normalized accuracy and weighted overall accuracy, information worth, FID and LPIPS (realism), detection mAP and recall.
    • Human evaluation: 30 participants on Amazon Mechanical Turk, each classifying 15 semantic counterexamples and rating realism on a 1–10 scale.
    • Repetitions: the paper does not report the number of experimental repetitions or variance.

    Main results

    Accuracy drop induced by each generation strategy on VGG-16 (overall column, per Table 1; benign is 0.986, lower means more effective attack):

    Strategyoverall accuracy
    Halton (large)0.274
    sCW0.485
    si-FGSM0.529
    Random (large)0.582
    sGD0.657
    Halton (small)0.757
    Random (small)0.830
    • The authors state that Halton (large) causes the largest accuracy drop, followed by sCW and si-FGSM; the drop is uneven across classes (class 0 airplane drops far less than class 3 bus).
    • The authors state that the trend on ResNet-50 differs slightly: si-FGSM causes the largest drop (overall 0.468, Table 14), and they believe hyperparameter choice affects method effectiveness.
    • The abstract states that semantic counterexamples can induce accuracy drops of "up to 50 percentage point" while remaining realistic.

    Robustness after augmentation

    • Retraining does not hurt generalization and improves robustness: after retraining VGG-16 with half of the training set replaced by counterexamples, benign accuracy is 0.978–0.98 (benign model 0.986), and accuracy on the si-FGSM/sGD/sCW test sets reaches 0.907–0.959 (Table 2).
    • Cross-method robustness transfer: regenerating counterexample tests on the augmented model (Table 5), the benign model is 0.483–0.685 under the three attacks, while the augmented model rises to 0.797–0.918; the authors state this suggests cross-method robustness transfer. The corresponding results for ResNet-50 in Appendix F.4 are weaker (e.g., the si-FGSM augmented model is 0.331 on sGDrobust, Table 19), which the authors do not explain.

    Realism and informativeness

    • FID/LPIPS: per-class FID of VGG-16 counterexamples is mostly between 2.48–38.27 and LPIPS between 0.12–0.67 (Table 4); the authors state the counterexamples are realistic under both metrics; a footnote notes that FID is mainly for generative models and may not be ideal for this purpose.
    • Human survey: participants' classification accuracy on counterexamples averages 98%; Redner samples have an average median realism of 6.67 (mean 6.35), and 3D-SDN samples have an average realism of 4.74 (median 4.26); the authors attribute the lower scores of the latter to poor rendering quality.
    • information worth: Halton (large) is highest (binary 1.1125), followed by sCW (0.9565) and si-FGSM (0.887) (Table 3); however, the authors state Halton sampling is sample-inefficient—the probability that each sample lowers accuracy is low, and obtaining one effective counterexample requires 5 times as many samples (Appendix E.5).

    Cross-model transfer and detection experiments

    • Classification transfer: counterexamples generated on VGG-16 transferred to ResNet-50 give overall accuracy sCW 0.588, si-FGSM 0.621, sGD 0.716 (Table 6); the authors state the transfer degree is lower than on the source model; the reverse direction (ResNet-50 → VGG-16) gives overall 0.509–0.761 (Table 18).
    • Detection (SqueezeDet): single-parameter modifications are largely ineffective (mAP 98.2–99.7, Table 7); the multi-parameter combo 2 (translate + rotate + mesh) lowers mAP from 99.4 to 65.9 (Table 8).
    • Detection robustness and transfer: after retraining with counterexamples, SqueezeDet achieves mAP 72.76 on counterexamples (a 7 percentage point improvement) and 84.73 on benign (15 percentage points lower than baseline) (Table 20); SqueezeDet counterexamples lower YOLOv3's mAP from 91.28% to 87.50%; the authors state detection counterexamples are most effective against the network that generated them, but do not rule out transfer.

    Other ablations and analyses

    • Parameter combinations (Redner): jointly modifying pose + vertex is more effective than single parameters (sCW: vertex 0.892, pose 0.543, joint 0.485, Table 9).
    • The lighting parameter does not affect classification accuracy and is omitted from all lists (Appendix E.2).
    • 3D-SDN counterexamples have average FID 0.102362 and average LPIPS 0.523521 (Appendix G.3).
  5. What are some points that can be further explored?

    The authors state that a future direction is to test semantically robust models in a CPS control loop; the experiments cover two renderers, four models, and two synthetic datasets.

    Limitations and future directions noted by the authors

    • Testing within the control loop: the conclusion (§5) states that future directions include testing the operation of these semantically robust models in a CPS control loop and making the algorithm aware of the rest of the control loop.
    • Semantic space distance metric: a footnote in §2.4 points out that the semantic parameter space is heterogeneous, and it is unclear whether a single function can characterize an appropriate distance; the norm should both measure semantic change and approximate human perception, and the "+" operation may be more complex for parameters such as weather and background.
    • Applicability of the FID metric: a footnote in §4.3 points out that FID is mainly used to measure the output quality of generative models and may not be ideal for this purpose.
    • 3D-SDN rendering quality: Appendix G.3 states that the low realism scores of 3D-SDN samples can be attributed to the poor rendering quality of this differentiable graphics framework.
    • Detection transfer not fully explored: Appendix G.5 states that multiple hyperparameters were not tested with 3D-SDN (counterexample generation is very time-consuming), so transferability in the detection task cannot be ruled out.
    • Limited code release: a footnote in §4.1 states that the corresponding implementation cannot be open-sourced due to licensing restrictions on the 3D-SDN code (part of the Redner code has been released).

    Scope of experiments

    • There are 4 models tested: VGG-16 and ResNet-50 (classification), SqueezeDet and YOLOv3 (detection; YOLOv3 is used only for transfer tests).
    • The datasets are two synthetic datasets, ShapeNet and VKITTI, plus KITTI used for training YOLOv3; no real-photo datasets are used.
    • The attack methods are sCW, si-FGSM, and sGD (detection uses only si-FGSM), with random and Halton sampling as baselines; all use 5 iterations.
    • Semantic parameters are limited to pose, vertices, color, weather, vegetation, translation, rotation, and mesh; lighting is omitted because it does not affect the results.
    • The paper does not report the number of experimental repetitions, confidence intervals, or statistical significance tests; the human survey sample size is 30 participants, 15 samples each.
  6. Summarize the main content of the paper

    Using differentiable rendering to move adversarial attacks into the semantic space to generate counterexamples; the authors claim they are realistic and informative, and that data augmentation with them can improve semantic robustness and transfer across models.

    An attack and enhancement paper on DNN semantic robustness: addressing the fact that pixel-level adversarial perturbations are not realizable in the real world, it proposes a general framework for generating semantic counterexamples in a semantic feature space (pose, color, lighting, etc.) via a differentiable renderer.

    • Method: i-FGSM, PGD variants, and CW are rewritten as si-FGSM, sGD, and sCW by the rule of "replacing the perturbation variable + applying the chain rule through the renderer for gradients"; an information worth metric combining classifier confidence and LPIPS realism is proposed to measure the value of enhanced samples.
    • Attack effectiveness: on ShapeNet, sCW counterexamples reduce VGG-16 overall accuracy from 0.986 to 0.485, and Halton large-range sampling reduces it to 0.274 but with low sample efficiency (about 5× the sampling cost); in detection, multi-parameter counterexamples reduce SqueezeDet's mAP from 99.4 to 65.9.
    • Realism: 30 participants classify counterexamples correctly at an average of 98%, and Redner samples have an average median realism of 6.67/10; under FID and LPIPS metrics the authors state the counterexamples are realistic.
    • Enhancement gains: after retraining on counterexamples, VGG-16 maintains 0.978–0.98 on benign and reaches 0.907–0.959 on the three attack test sets, and is also robust to unseen attack methods (0.797–0.918).
    • Transferability: counterexamples generated on VGG-16 transfer to ResNet-50 (overall 0.588–0.716); detection counterexamples transfer weakly to YOLOv3 (mAP 91.28% → 87.50%).
    • Authors' conclusion: semantic counterexamples are realistic and informative, and using them for data augmentation can improve semantic robustness without harming generalization; future directions include testing robust models in CPS control loops.

Related papers

Read originalarxiv.org