Skip to content

CORGI: Providing certifiable interpretability robustness guarantees for class activation maps (CAM)

Certified Interpretability Robustness for Class Activation Mapping

arXiv· arXiv:2301.11324· Alex Gu, Tsui-Wei Weng, Pin-Yu Chen, Sijia Liu, Luca DanielMIT; MIT-IBM Watson AI Lab; IBM ResearchIndexed by this site · 23 days ago· Originally publishedAlignment

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

The authors propose CORGI, which gives a certifiable robustness lower bound for the top-k pixels of CAM interpretability maps; on GTSRB, the median ratio of the certified radius to the attack upper bound is 4.7–5.0.

Threat modelThe attacker applies an L∞-norm-bounded, visually imperceptible perturbation to the input image, with the goal of changing the top-k pixel set of the CAM interpretability map (the network classification may remain correct); the defender uses CORGI to give a certified radius r, guaranteeing that the top-k pixel set remains unchanged when ||x−x'||∞≤r.

Questions
Safety-critical scenarios such as autonomous driving rely on the interpretability of deep networks, but explanation methods such as CAM can be changed by small adversarial perturbations, while previously there was almost no certifiable guarantee for interpretability robustness.
Methods
The authors propose CORGI: use CNN-Cert to compute upper and lower bounds on the last convolutional layer output under perturbation, synthesize upper and lower bounds for each pixel of the CAM map, then binary-search for the maximum certifiable radius satisfying the condition that the top-k pixel set remains unchanged, and give a k2 relaxed version and a theoretical certificate based on the Lipschitz constant.
Experiments and results
On GTSRB, using a self-trained four-layer CNN, comparing the CORGI lower bound with the Ghorbani attack upper bound on 100 images each of class 12 and class 17: the median ratios are 4.7 and 5.0 respectively; the larger the CAM gap, the larger the certificate and the attack bound.
Limitations and future work
The authors note that CORGI performs well only on small networks, taking about 1 minute per image, and cannot yet be extended to the large networks of real autonomous driving; and that it covers only CAM and does not handle methods such as GradCAM, Integrated Gradients, and DeepLIFT.
Experimental setupsmall CNN (4 conv layers, GTSRB) · GTSRB · certified radius r、attack upper bound ε et al.
Evaluated models
small CNN (4 conv layers, GTSRB)
Benchmarks
GTSRB
Metrics
certified radius rattack upper bound εratio ε/r
AI summaryThe researchers propose the CORGI algorithm, which provides certifiable robustness guarantees for class activation map (CAM) interpretability maps. In view of the current situation that most interpretation methods for autonomous driving perception models have been shown to lack robustness and few are certified, the algorithm gives a certifiable lower bound on the robustness of the top k pixels in the CAM of an input image. In a case study on traffic sign data, its certified lower bound on the minimum adversarial perturbation is within a factor of 4-5 of the state-of-the-art attack method.

The researchers propose the CORGI algorithm, which provides certifiable robustness guarantees for class activation map (CAM) interpretability maps. In view of the current situation that most interpretation methods for autonomous driving perception models have been shown to lack robustness and few are certified, the algorithm gives a certifiable lower bound on the robustness of the top k pixels in the CAM of an input image. In a case study on traffic sign data, its certified lower bound on the minimum adversarial perturbation is within a factor of 4-5 of the state-of-the-art attack method.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Provide a certifiable adversarial robustness lower bound for the top-k pixels of CAM interpretability maps, a direction that was previously almost blank.

    Core question: how to give a certifiable robustness guarantee for the top-k pixels of CAM interpretability maps under adversarial perturbation.

    • Scenario and motivation: the authors believe that safety-critical systems such as autonomous driving make extensive use of deep perception models, and explaining their predictions is important for building human-machine trust; however, existing work has shown that explanation methods can also be broken by visually imperceptible perturbations—the network classification remains correct, but the explanation map becomes unreasonable.
    • Existing gaps: the authors state that adversarial robustness certification for classifiers has been well studied, but robustness certification for interpretability methods has been done very little; the closest work, Levine et al. (2019), certifies a modified version of SmoothGrad, not CAM.
    • Core observation: CAM requires only a forward pass and can localize class-discriminative regions, which the authors consider better suited to autonomous driving scenarios that must distinguish similar traffic signs; the top-k pixel set remaining unchanged under perturbation can serve as a measure of explanation robustness.
    • The paper proposes: CORGI (Certifiably prOvable Robustness Guarantees for Interpretability mapping), which the authors call the first algorithm to give a robustness lower bound for CAM interpretability, accompanied by three theoretical results: exact top-k, relaxed top-k, and one based on the Lipschitz constant.
    • Threat model:
      • Attacker goal: change the top-k pixel set of the CAM map under an L∞-norm-bounded perturbation, while the classification result may remain unchanged.
      • Attacker knowledge: the attack used as the upper bound in the paper (Ghorbani et al., 2019) is a gradient method and requires access to the network (gradients computed after replacing ReLU with softplus).
      • Defender capability: the defender has the network weights and uses CNN-Cert to compute layer-wise bounds and give a certified radius r.
      • Victim system: a convolutional network with a global average pooling layer (the structure required by CAM); the paper's experiment is a GTSRB traffic sign classifier.
  2. What related research exists?

    Related work falls into four groups: adversarial attacks, certified defenses, interpretability methods and their fragility, and most recently Levine et al.'s certification of SmoothGrad.

    Adversarial attacks

    • Szegedy et al. (2013) pointed out that deep networks are not robust to adversarial perturbations; Guo et al. (2019) extended attacks to the black-box setting.
    • FGSM (Goodfellow et al., 2014), PGD (Kurakin et al., 2016), and Carlini-Wagner (Carlini and Wagner, 2017) are the progressively stronger attack sequence listed in the paper.

    Certified defenses (for classification)

    • The paper lists three classes of methods that approximate the maximum certified radius: semidefinite programming (Raghunathan et al., 2018), mixed integer programming (Tjeng et al., 2017), and linear bounds on activation functions (Boopathy et al., 2019, i.e., CNN-Cert); Weng et al. (2018) pointed out that finding the maximum certified radius is hard even in the approximation-ratio sense.
    • Singh et al. (2019) and Zhang et al. (2018) are cited as representatives of classifier robustness certification.

    Interpretability methods and their fragility

    • CAM (Zhou et al., 2016), GradCAM (Selvaraju et al., 2017), Integrated Gradients (Sundararajan et al., 2017), SmoothGrad (Smilkov et al., 2017), and DeepLIFT (Shrikumar et al., 2017) all generate per-pixel explanation maps.
    • All three works Zheng et al. (2019), Alvarez-Melis and Jaakkola (2018), and Ghorbani et al. (2019) successfully generated imperceptible perturbations that keep the classification correct but make the explanation map unreasonable.

    Certified explanation robustness

    • Levine et al. (2019) certified the explanation robustness of a modified version of SmoothGrad and is the work the paper names as most relevant; the paper follows its top-k metric but turns to CAM, on the grounds that CAM can localize class-discriminative regions, is better suited to distinguishing similar traffic signs, and requires only a forward pass whereas SmoothGrad requires one backward pass per test image.

    Baselines and benchmarks: the paper has no baseline certified method (the authors state that CORGI is the first CAM explanation robustness certification framework); the comparison object is the Ghorbani et al. (2019) iterative feature importance attack as an upper bound; the experimental benchmark is GTSRB (Stallkamp et al., 2012). The authors position this paper as, building on Levine et al.'s top-k certification idea, the first to cover CAM, an explanation method better suited to autonomous driving scenarios.

  3. How does the paper address this problem?

    CORGI uses CNN-Cert to compute per-pixel upper and lower bounds of the CAM map, then binary-searches for the maximum certifiable radius satisfying the top-k unchanged condition.

    Overall approach: first use CNN-Cert to compute bounds on the last convolutional layer's output under a perturbation radius δ, combine them with the fully connected layer weights to compose per-pixel lower and upper bounds L(δ), U(δ) of the CAM map, then binary-search for the largest δ that guarantees the top-k pixel set remains unchanged, as a lower bound on the minimum attack radius r*. CORGI also includes a relaxed version of the certificate and a theoretical certificate based on the Lipschitz constant.

    Problem setting and CAM formalization

    • For an input x and predicted class c, the CAM map I(x) is a weighted sum of the last convolutional feature maps: Mcx,y = ∑k wck fkx,y, where f^k is the output of the k-th filter and w^c_k is the fully connected layer weight.
    • Robustness goal: the set of top-k pixel positions remains unchanged after perturbation, i.e. {σ1,…,σk} = {σ'1,…,σ'k}, with no requirement that the internal order be consistent.
    • The authors note that the CORGI bound does not guarantee the classifier itself is robust, but one can take the minimum of the CORGI bound and the classification robustness bound; in the experiments the CORGI bound is almost always smaller than the classification robustness bound.

    Computing CAM bounds with CNN-Cert

    • First use CNN-Cert to compute per-pixel bounds l^k(δ) ≤ f^k(x') ≤ u^k(δ) for each filter of the last convolutional layer's output.
    • Split the sum according to the sign of the weights: negative weights multiply the upper bounds and positive weights multiply the lower bounds to obtain the CAM lower bound L(δ), and vice versa for the upper bound U(δ) (Eq. 1–3).

    Certification condition and binary search

    • Theorem 3.2: if mink′ ∈ 1..k L(δ)σk′ ≥ maxk′ ∈ k+1..s2 U(δ)σk′, then δ is a certified lower bound on r*, i.e. the top-k set is unchanged.
    • Since the CNN-Cert bounds are monotone in δ, Algorithm 2 binary-searches over [0,1] for P rounds and returns the largest δ satisfying the condition.
    • Relaxed version (Theorem 3.3): it only requires the original top-k pixels to remain within the top-k2 of the perturbed map (k2 ≥ k), and the condition becomes the first k terms of min L(δ) ≥ U(δ){k2−k+1}; when k2 = k it reduces to the original condition.

    Lipschitz constant certificate (Appendix 8.3)

    • Define g_p^(i)(x) = I(x)σi − I(x)p, and use its local Lipschitz constant to give the certified radius r = min over (i,p) of g_p^(i)(x)/L_p^(i)(δ).
    • The authors believe that when the Lipschitz constant estimate is sufficiently tight this bound may be better than CORGI, but because existing estimation methods are either loose or slow, no empirical evaluation was done.

    Attack upper bound (for comparison)

    • Reproduce the iterative feature importance attack of Ghorbani et al. (2019): targeting the original top-k pixel set B, use projected FGSM to minimize the sum of the values at these positions in the new image; replace ReLU with softplus to compute gradients.
    • The paper makes one modification: it returns the y with the fewest pixels overlapping the original image's top-k, rather than the y with the greatest dissimilarity, because that is what CORGI certifies.
  4. What experiments did the paper conduct?

    On a self-trained four-layer CNN on GTSRB, comparing the CORGI lower bound and the attack upper bound on 100 images from each of two classes, the median ratio is 4.7–5.0.

    Experimental setup

    • Target model: a self-trained small CNN—4 convolutional layers (8, 8, 16, 16 filters) + global average pooling + a fully connected layer (the structure required by CAM), CAM map size 24×24, GTSRB test accuracy about 92% (Appendix 8.1).
    • Dataset: GTSRB, stated in the paper as more than 50000 48×48×3 traffic sign images in 43 classes.
    • Comparison target: the iterative feature importance attack of Ghorbani et al. (2019) (Algorithm 1), P = 300 steps, α = ε/10; the appendix states that increasing P has little effect on attack effectiveness.
    • Metrics: the CORGI certified radius r (a lower bound on r*), the smallest successful perturbation ε found by the attack (an upper bound on r*), and the ratio ε/r of the two.
    • Sample size: the main experiment uses 100 randomly selected images each from class 17 (no entry) and class 12 (priority road), 200 in total; Appendix 8.5 additionally has 100 images each from 5 classes.
    • Number of repetitions: not reported in the paper.

    Main results

    Single example: at k = 15, the CORGI certified radius of one no entry image is r = 0.0009; Figure 1 shows the perturbation images, CAM maps and top-k positions at r and two larger ε, and the top-k pixels remain unchanged within the certificate range.

    Swipe horizontally to read the table

    ClassNumber of samplesMedian ε/rSamples retained in histogram
    class 17 (no entry)1005.092/100
    class 12 (priority road)1004.799/100

    According to Figure 2c and the main text. The histogram excludes outlier samples with a ratio above 20.

    • On this basis the authors state that the CORGI lower bound differs from the attack upper bound by only a factor of 4–5, and is therefore not far from the optimal perturbation r*.
    • Figure 2b gives the CORGI lower bound and attack upper bound for each of the 200 images (sorted by CORGI lower bound); the main text gives no specific values.
    • Appendix 8.5 repeats this comparison on 100 images each from another 5 classes; the authors state that the ratio distributions are similar, indicating that CORGI performs uniformly on the test distribution; the specific values in Figure 5 are not legible in the text.

    CAM gap and the choice of k

    • What was tested: k varied from 1 to 50, examining the relationship of the CAM gap (I(x)[k] − I(x)[k+1]) with the CORGI certificate and the attack bound.
    • Results: Figure 3b shows that as the CAM gap increases, both the CORGI certificate and the attack bound increase; the main text gives no specific values.
    • Authors' interpretation: the top-k maps corresponding to k with a large CAM gap are more robust, which can serve as a basis for choosing k—too small a k may miss salient regions, while too large a k may include noise pixels.

    Relaxing the top-k condition

    • What was tested: with the image and k fixed, k2 was varied (the maximum position into which the original top-k pixels are allowed to fall in the perturbed image).
    • Results: Figure 3c shows that when k2 is close to k, a small increase in k2 can markedly increase the certificate; as k2 continues to increase, the CORGI bound levels off. The main text gives no specific values, and the authors' explanatory sentence for this phenomenon is unfinished in the text (nothing follows "We believe that this occurs because").

    Other ablations and analyses

    • Attack hyperparameters: P = 300, α = ε/10; Appendix 8.1 states that increasing P has no significant effect on attack effectiveness (the paper uses the word significant here and reports no statistical test).
    • Lipschitz certificate (Theorem 8.3): a theoretical result only; the authors explicitly do not evaluate it empirically, because existing methods for estimating the Lipschitz constant are either too loose or too slow.
    • Appendix 8.4 gives two further sets of examples similar to Figure 1, with CORGI bounds and CAM maps.
  5. What are some points that can be further explored?

    The authors state that CORGI only applies to small networks, takes about 1 minute per image, and only covers CAM; the experiments are conducted only on a self-trained small CNN on GTSRB.

    Limitations and future directions noted by the authors

    • Performs well only on small networks (Conclusions and Future Work): even for small networks, CORGI requires about 1 minute per image.
    • Cannot yet scale to real applications (Conclusions and Future Work): GTSRB image resolution is lower than what autonomous driving actually sees, and real networks are much larger; the authors state that CORGI cannot yet scale to real scenarios.
    • The accuracy bottleneck lies in bounds for large networks (Conclusions and Future Work): the authors state that the main bottleneck is the lack of fast and tight bounds for large convolutional networks, and that improving layer-by-layer network bounds and explanation bounds on large networks is a key direction.
    • Covers only CAM (Conclusions and Future Work): DeepLIFT, Integrated Gradients, and GradCAM are not yet handled; the authors note that CAM imposes requirements on network architecture (a global average pooling layer must follow the convolutional layers), and that building bounds for other explanation methods could expand the range of certifiable models.
    • The Lipschitz route is not empirically tested (Appendix 8.3): the authors believe that Theorem 8.3 could outperform CORGI if the Lipschitz constant estimate is tight enough, but existing estimation methods are either very loose or very slow, so no experimental analysis was performed.

    Scope of experiments

    • The model under test is a self-trained four-layer CNN (8, 8, 16, 16 filters + global average pooling), with about 92% test accuracy on GTSRB (Appendix 8.1).
    • The dataset is GTSRB only; the main experiments cover 100 images each from class 12 and class 17, and Appendix 8.5 additionally covers 100 images each from 5 classes.
    • The comparison upper bound is only the Ghorbani et al. (2019) attack (P = 300, α = ε/10); the paper does not report comparisons with other certification methods (the authors state that CORGI is the first CAM explanation certification framework).
    • The paper does not report the number of experimental repetitions, nor does it report a comparison of the respective runtimes of CORGI and the attack (only that CORGI takes about 1 minute per image).
    • The experiment on the effect of k varies k (1 to 50) and k2 on a single image; the specific values in Figure 3b and 3c are not legible in the text.
  6. Summarize the main content of the paper

    CORGI gives a certifiable robustness lower bound for the top-k pixels of CAM explanation maps, with a median ratio to the attack upper bound of 4.7–5.0 on GTSRB.

    An interpretability robustness certification paper: addressing the problem that in scenarios such as autonomous driving, CAM explanation maps can be broken by small adversarial perturbations while previously there was a lack of certification guarantees, it proposes CORGI.

    • Method: use CNN-Cert to compute per-pixel bounds on the last convolutional layer's output under perturbation, combine them with the fully connected layer weights to compose upper and lower bounds of the CAM map, then binary-search for the maximum radius that guarantees the top-k pixel set remains unchanged; it also gives a k2 relaxed version of the certificate and a theoretical certificate based on the Lipschitz constant (the latter not empirically evaluated).
    • Comparison setup: the iterative feature importance attack of Ghorbani et al. (2019) is used as the upper bound (P = 300, α = ε/10), on a self-trained four-layer CNN on GTSRB (test accuracy about 92%).
    • Main results: on 100 images each from class 17 and class 12, the median ratios of the attack upper bound to the CORGI lower bound are 5.0 and 4.7 respectively (Figure 2c); for a single no entry image at k=15 the certified radius is r = 0.0009; the larger the CAM gap, the larger the certificate and the attack bound (Figure 3b); the ratio distributions for the 5 additional classes in the appendix are similar.
    • Authors' conclusion: the authors state that CORGI is the first robustness certification algorithm for CAM interpretability, that the lower bound is only 4–5 times away from the attack upper bound, and that it can serve as a baseline for future work.
    • Limitations noted by the authors: it performs well only on small networks (about 1 minute per image) and cannot yet be extended to the large networks of real autonomous driving; it covers only CAM, leaving GradCAM, Integrated Gradients, DeepLIFT, etc. for future work.

Related papers

Show 3 more papers
Read originalarxiv.org