How cross-architecture knowledge distillation affects model interpretability: a study of visual explanation transfer from ResNet-152 to ResNet-34

When knowledge distillation compresses a large network into a smaller one, does it transfer spatial feature attribution along with it, and which hyperparameters and architectures determine this.

• Scenario: the authors state that compact convolutional networks can approach the classification accuracy of much larger models, making them suitable for deployments constrained by compute, memory, or latency; but once smaller, the features driving predictions become harder to see. The authors state this is especially important in real-time perception for autonomous driving and medical diagnostic support.
• Existing gap: KD from Hinton et al. lets a lightweight student reproduce a large teacher's temperature-softened output distribution, thereby learning inter-class relationships rather than just ground truth. The authors state that most KD work mainly reports classification accuracy, and whether students preserve behavior that makes decisions interpretable remains unclear.
• Cannot extrapolate from pruning: pruning removes parameters from an existing architecture, while distillation trains a new smaller architecture to mimic the teacher. The authors state that spatial interpretability conclusions from pruning cannot be directly extrapolated, and it is unclear whether such transfer depends on the balance between soft labels and ground truth.
• What this paper does: applies the Grad-CAM, RMA, and RRA framework originally used for pruning to KD. On ImageNet-1K, distills ResNet-152 into ResNet-34, sweeping temperature T and soft-label weight α; then uses VOC2012 object masks to measure heatmap alignment. Also evaluates 12 cross-architecture combinations of convolutional networks and transformers. The authors state that, to their knowledge, this is the first use of this interpretability evaluation framework for KD.

The paper discusses in the introduction and Related Work mainly visual explanation, interpretability-oriented pruning, and standard logit distillation.

Visual explanation

• Grad-CAM (Selvaraju et al., 2017): generates heatmaps from convolutional layer gradients, highlighting image regions important to the prediction. The paper uses it as input for quantitative evaluation without additional criticism.
• RMA and RRA (Arras et al., 2022): the paper states that RMA is the proportion of total heatmap relevance falling within the ground-truth bounding box, and RRA evaluates this relationship using only the highest-relevance pixels. The paper adopts these two metrics without criticizing their definitions.

Interpretability in pruning

• Cassano et al. (2025): uses Grad-CAM with RMA and RRA to compare pruned convolutional networks and vision transformers. The authors relay their observation: when removing parameters, convolutional networks typically trade off accuracy and interpretability, while vision transformers show a tighter correlation between the two.
• Yeom et al. (2021): uses Layer-wise Relevance Propagation (LRP) during training to identify and prune less relevant components, allowing interpretability and compression to be optimized together. The paper does not criticize this pruning criterion.

Knowledge distillation

• Hinton et al. (2014): the student reproduces the output distribution of a larger, pretrained teacher; temperature softens the probabilities so the student learns inter-class relationships rather than just fitting ground-truth labels. The introduction states that most KD literature focuses on classification accuracy.

Baselines and benchmarks

• Baseline: the undistilled student, i.e., the "-" row in Table 1 and the Student Baseline in Table 2; Table 1 also reports the three teachers separately. The paper does not use other distillation algorithms as comparisons.
• Benchmarks: top-1 and top-5 on ImageNet-1K; pixel-level segmentation masks from VOC2012 for RMA and RRA.

The authors' positioning is that pruning and distillation have different compression mechanisms: the former removes parameters from an existing architecture, while the latter trains a new smaller architecture to mimic the teacher, so pruning conclusions cannot be directly extrapolated. The authors apply the above framework to KD to see whether students inherit the teacher's spatial localization decisions and whether this depends on the weight of soft labels versus ground truth; they state that, to their knowledge, this is the first use of this framework for KD.

The overall approach is standard logit Knowledge Distillation: a frozen large teacher provides soft labels, and the small student simultaneously fits the ground truth; after training, alignment of heatmaps and object masks is used to see whether spatial explanations are preserved.

Data, Architecture, and Pairing

• Data: ImageNet-1K is used for all training, and top-1 and top-5 are evaluated for both teacher and student. VOC2012 provides pixel-level segmentation masks to define the spatial boundaries of objects.
• Main experiment: Teacher ResNet-152, student ResNet-34. The authors state that the parameter count is roughly compressed by three times; the paper does not give exact parameter counts.
• Cross-architecture: Teachers are ResNet-152, EfficientNet, ViT; students are ResNet-34, MobileNetV4, lightweight EfficientNet, compact ViT. Table 1 lists efficientnet_b7, vit_l_16, efficientnet_b0, vit_s_16, mobilev4 small. Among the 12 pairs, there are CNN-to-CNN, Transformer-to-Transformer, and heterogeneous pairs in both directions.

Distillation Objective

• The teacher is frozen. Temperature T is applied to logits before softmax. The authors state that the larger T, the flatter the distribution, making it easier for the student to see the probabilities of secondary classes.
• The training objective is L=(1−α)L_ce(y,p_s)+α T²L_KL(σ(z_t/T),σ(z_s/T)). y is the ground truth, p_s is the student's predicted distribution, z_t and z_s are the teacher's and student's logits, σ is softmax. α is between 0 and 1, adjusting the weight of the KL term for soft labels relative to the cross-entropy for hard labels; the square of T is used to make the gradient magnitude of the soft-label term comparable to that of the hard-label term.
• The loss given in the main text has only these two terms; no feature map or intermediate layer alignment is mentioned.

Explanation Metrics

• Grad-CAM uses gradients flowing into the final convolutional layer to produce a coarse localization map. Guided Grad-CAM further combines with guided backpropagation to obtain higher-resolution pixel-level heatmaps.
• Heatmap H is channel-pooled and min-max normalized; mask M is thresholded at 0.5. RMA is the sum of normalized heatmap mass inside the mask divided by the total mass plus a small constant ε (the paper gives machine precision as an example). RRA is the proportion of the top N pixels in the heatmap that fall inside the mask, where N is the number of mask pixels. Both metrics theoretically range from 0 to 1; higher values indicate better alignment with the human-annotated object region.
• Thus there are GC Mass, GC Rank, GGC Mass, GGC Rank. Related Work describes RMA as the proportion falling within the bounding box, while the Methods section uses segmentation masks; the paper does not explain how the two correspond. The paper does not set a numerical threshold for what counts as an acceptable explanation.

Training Procedure and Hyperparameters

• Figure 1: The frozen teacher's soft labels and the ground truth both enter Equation 1. Then top-1 is computed on the ImageNet-1K validation set, and RMA and RRA are computed on VOC2012.
• The student is initialized with ImageNet pretrained weights, optimizer is SGD, initial learning rate 0.001, schedule is cosine-linear, total 15 epochs, batch size 128, training uses AMP. At the end of each epoch, prediction performance is checked on the ImageNet-1K validation set. Implementation uses PyTorch, logging uses Weights & Biases.
• Hyperparameter sweep uses T=4, α=0.5 as reference: with α fixed at 0.5, T is changed to 1 and 10; with T fixed at 4, α is changed to 0.1 and 0.9. Including the reference group, there are five groups, corresponding to Table 2. The paper does not specify the T and α for each of the 12 cross-architecture pairs.

Experimental Setup

• Models under test: Teachers ResNet-152, EfficientNet-B7 (efficientnet_b7 in table), ViT-L/16 (vit_l_16); students ResNet-34, EfficientNet-B0, ViT-S/16 (vit_s_16), MobileNet-V4-Small (mobilev4 small). The main text writes MobileNetV4, but reference [7] is titled MobileNetV2.
• Data: ImageNet-1K for training and Acc1, Acc5; VOC2012 masks for explanation metrics. The paper does not report how many images were used for explanation evaluation.
• Training: Students initialized from ImageNet pretrained weights, SGD, initial learning rate 0.001, cosine-linear, 15 epochs, batch size 128, AMP. Five T, α settings are in Table 2. The T and α for the 12 cross-architecture pairs are not specified in the paper.
• Metrics: Acc1, Acc5; RMA (Mass) and RRA (Rank) for Grad-CAM and Guided Grad-CAM. Percentages in parentheses are changes relative to the undistilled student in the table. No LLM evaluation, and no human agreement numbers.
• Baselines: Each student's undistilled result. Table 1 also has separate rows for the three teachers. In Table 2, teachers only have Acc1, with attribution left blank.
• The two tables should not be merged: The resnet152→resnet34 in Table 1 and the five groups in Table 2 are not the same row of numbers. The paper does not report number of repetitions or statistical tests.

Main Results

The table below is from Table 2, with teacher ResNet-152 and student ResNet-34. The abstract writes the top-1 for these five groups as 71.6% to 74.0%, Grad-CAM RMA as 7.7% to 9.7%, RRA as 7.3% to 10.1%, Guided Grad-CAM RMA as 16.1% to 18.6%, RRA as 15.9% to 21.5%. Teacher Acc1 is 82.284, attribution not reported, so not included. Config 4's Acc1 relative change is written in the table as (+1,0%), while the main text writes +1.0%, 74.02%.

Swipe horizontally to read the table

Config	Acc1	GC Mass	GGC Mass	GC Rank	GGC Rank
Student baseline	73.314	0.0915	0.2027	0.1002	0.2041
T=1, α=0.5	73.953 (+0.9%)	0.0969 (+5.9%)	0.1853 (-8.6%)	0.0993 (-0.9%)	0.2149 (+5.3%)
T=4, α=0.5	73.346 (+0.0%)	0.0822 (-10.2%)	0.1808 (-10.8%)	0.0933 (-6.9%)	0.1717 (-15.9%)
T=10, α=0.5	73.369 (+0.1%)	0.0839 (-8.3%)	0.1854 (-8.5%)	0.0955 (-4.7%)	0.1813 (-11.2%)
T=4, α=0.1	74.023 (+1.0%)	0.0952 (+4.0%)	0.1864 (-8.0%)	0.1005 (+0.3%)	0.2134 (+4.6%)
T=4, α=0.9	71.631 (-2.3%)	0.0767 (-16.2%)	0.1612 (-20.5%)	0.0733 (-26.8%)	0.1586 (-22.3%)

• α and T: The authors believe that the soft-label weight α is more decisive than temperature T in whether interpretability is preserved; leaning more on hard labels preserves accuracy and coarse localization, while α=0.9 degrades accuracy and all four attribution metrics together.
• Temperature summary: The authors state that with α fixed, increasing T tenfold moves each metric by less than two percentage points, and less than changing α from 0.5 to 0.9. Table 2 does not give that difference; T=1 and T=10 GGC Rank are 0.2149 and 0.1813.
• Fine-grained: All five groups' GGC Mass are below the undistilled baseline. The authors state that even the best setting is about 8% lower (Config 4 in the table is -8.0%), and believe logit distillation more easily transfers "where to look" rather than pixel-level structure. GGC Rank in Config 1 and Config 4 is above baseline, so "fine-grained all below baseline" matches the table only for GGC Mass.

Cross-Architecture Distillation

Table 1 is 3 teachers × 4 students. The table below lists only Acc1; teacher column order matches the table. The authors state that ViT students rise from 74.63% to about 80.8% under two convolutional teachers; CNN students do not have equivalent gains. resnet34 under efficientnet_b7 is +0.2%, not all below baseline.

Swipe horizontally to read the table

Student	Baseline Acc1	resnet152	efficientnet_b7	vit_l_16
resnet34	73.31	73.27 (-0.1%)	73.47 (+0.2%)	73.15 (-0.2%)
efficientnet_b0	77.69	77.25 (-0.6%)	77.18 (-0.7%)	77.10 (-0.8%)
vit_s_16	74.63	80.82 (+8.3%)	80.83 (+8.3%)	79.50 (+6.5%)
mobilev4 small	73.73	73.49 (-0.3%)	73.39 (-0.5%)	72.48 (-1.7%)

• Coarse/fine divergence: The authors state that most students' Grad-CAM rises, while some CNNs' Guided Grad-CAM falls. resnet34's GC Mass goes from 0.063 to 0.083, 0.083, 0.082; GC Rank from 0.049 to 0.094 (+91.8%), 0.097 (+98.0%), 0.089 (+81.6%); GGC Mass from 0.2284 to 0.1851, 0.1870, 0.1830; GGC Rank is -26.1%, -23.1%, -26.5%. mobilev4 small's GGC Mass from 0.1589 to 0.1250, 0.1230, 0.1230.
• EfficientNet-B0 exception: GC Mass from 0.086 to 0.070, 0.077, 0.073; GGC Mass from 0.0450 to 0.0600, 0.0590, 0.06. The discussion states that convolutional students' fine-grained attribution consistently worsens; this student's GGC Mass and GGC Rank rise in Table 1, so the table is authoritative.
• ViT-S: Under ResNet-152, all four attribution metrics rise (GGC Mass 0.0690 to 0.0870, GGC Rank 0.084 to 0.087). Under EfficientNet-B7, GGC Rank is 0.083 (-1.2%). Under vit_l_16, GC Mass 0.100 (+92.3%), GC Rank 0.150 (+108.3%), GGC Mass 0.0320 (-53.6%), GGC Rank 0.037 (-56.0%). The authors state that ViT-S can improve both coarse and fine attribution; the fine-grained cells for the latter two teachers do not all rise. The authors believe inheritance is mainly limited by student architectural bias, not teacher complexity.

Qualitative Heatmaps

• Figure captions: The authors state that Figures 2–4 show comparison heatmaps between baseline and distilled, including coarse localization gains and fine-grained trade-offs. The caption says Figure 3 shows improved coarse localization after distillation, Figure 4 shows worsened fine-grained attribution, Figure 2 compares baseline ResNet-34 with ResNet-152 to ResNet-34. The converted text has no numerical values or axes from the figures, and does not describe heatmap locations.

Other Ablations and Analyses

• Table 1 teacher rows: resnet152 Acc1 78.31, Acc5 94.05, GGC Mass 0.339; efficientnet_b7 Acc1 84.12, Acc5 96.91, GGC Mass 0.045; vit_l_16 Acc1 79.66, Acc5 94.63, GGC Mass 0.1, GGC Rank 0.038.
• Table 2 teacher Acc1 is 82.284, different from Table 1's resnet152 Acc1 78.31; the paper does not explain.
• resnet34 baseline in the two tables: Table 1 GC Mass 0.063, GGC Mass 0.2284; Table 2 0.0915, 0.2027. The paper does not explain.
• vit_s_16 Acc5 from 92.67 to 95.92 (+3.5%), 95.85 (+3.4%), 94.90 (+2.4%), teacher order is resnet152, efficientnet_b7, vit_l_16 (Table 1).
• The authors state that ViT-L to EfficientNet-B0 GGC Mass falls in 0.058–0.060; in the table under the three teachers it is 0.0600, 0.0590, 0.06.
• The main text writes that same-architecture ViT-L to ViT-S provides a reference for representation transfer with structural matching; the sentence ends at the semicolon, with no further conclusion.

Limitations and future directions noted by the authors

• Interpreters and tasks: The authors state that the evaluation relies on gradient-based visual interpreters (Grad-CAM variants) and image classification benchmarks (Limitations and Future Work).
• Gradient saturation: The authors state that gradients may saturate, and future work should add perturbation-based methods (e.g., SHAP) or propagation-based methods (e.g., LRP) (same section).
• Dense prediction: The authors suggest examining how spatial reasoning transfers under compression on dense prediction tasks such as object detection (same section).

Experimental coverage

• The models in the result tables are ResNet-152, EfficientNet-B7, ViT-L/16, ResNet-34, EfficientNet-B0, ViT-S/16, MobileNet-V4-Small. Table 1 has 12 paired groups, and Table 2 has 5 groups of T and α from ResNet-152 to ResNet-34.
• Accuracy is reported as Acc1 and Acc5 on ImageNet-1K; interpretability metrics are RMA and RRA for Grad-CAM and Guided Grad-CAM, compared against VOC2012 masks.
• The Training Protocol states that students use ImageNet pretrained weights, SGD, initial learning rate 0.001, cosine-linear scheduling, 15 epochs, batch size 128, and AMP.
• The paper does not report the number of images used for attribution on VOC2012, repetitions, random seeds, or statistical tests, nor does it report T and α across the 12 cross-architecture groups.
• The paper does not specify which layer Grad-CAM uses for ViT (the Methods section says the final convolutional layer), nor does it explain why ResNet-152's Acc1 (78.31 vs. 82.284) and the ResNet-34 baseline attribution scores differ between Table 1 and Table 2.

The authors use classification accuracy and alignment of heatmaps with masks to check whether logit distillation transfers the large network's spatial explanations to a smaller student.

• Problem: Compact models are convenient for deployment where compute or latency is constrained. The authors claim that after shrinking, the decision basis is harder to see, and autonomous driving perception and medical diagnosis support need this basis. Accuracy alone cannot answer whether the student inherits the teacher's spatial attention.
• Method: On ImageNet-1K, the loss mixes hard-label cross-entropy and temperature-softened teacher distribution by α. The student starts from ImageNet pretrained weights and is trained with SGD for 15 epochs. The main sweep is five T, α pairs from ResNet-152 to ResNet-34; there are also 12 pairs with 3 teachers and 4 students. Explanations use Grad-CAM and Guided Grad-CAM, and RMA, RRA are computed relative to VOC2012 masks.
• Hyperparameters: In Table 2, Acc1 is 74.023% at T=4, α=0.1, and 71.631% at T=4, α=0.9. At α=0.9, the relative changes for all four attribution metrics are negative, with GC Rank at -26.8%. All five GGC Mass values are below the undestilled baseline of 0.2027.
• Cross-architecture: In Table 1, vit_s_16's Acc1 goes from 74.63 to 80.82 (+8.3%) under the ResNet-152 teacher, and all four attribution metrics rise in that row. The same student under vit_l_16 has Acc1 of 79.50 (+6.5%), with GGC Rank at -56.0%. EfficientNet-B0's GGC Mass rises and GC Mass falls, opposite to ResNet-34.
• Authors' conclusion: How much spatial explanation survives depends on how the distillation objective is configured; α is more critical than T. Object-level attention can be preserved or slightly increased at low α, while pixel-level attribution is harder to retain. The authors claim that if hard labels are prioritized, temperature scaling is kept low, and Vision Transformer's representational flexibility is used, the spatial reasoning that makes the network transparent and trustworthy can be preserved.