Representational geometry as a diagnostic tool for adversarial robustness
Representational Geometry as a Diagnostic for Adversarial Robustness
The authors use M-PHATE embeddings and geometric metrics to analyze 48 MNIST MLPs, and state that adversarial training reshapes representation geometry, and that geometric stability and robust accuracy rank defenses in opposite orders.
- Robustness is usually reported only as a single scalar, attacked accuracy, which cannot reflect how the network's internal representation structure is changed by adversarial training; when multiple networks have similar accuracy, their representation geometries may be completely different.
- Train 48 three-layer MLPs on MNIST (vanilla plus three single-step adversarial training objectives, five perturbation budget levels), use M-PHATE to embed the penultimate-layer representations of test samples into a low-dimensional space, and measure the normalized displacement, MMD permutation test, and kNN recovery rate of clean and perturbed points.
- The authors report: by accuracy ranking under attacks matched to the threat model, the l∞-trained network is the most robust; but the geometric metrics rank it last (displacement 0.233, kNN recovery 0.30 under Extreme High), the FGSM-trained network is geometrically the most stable, and the divergence widens as the budget increases.
- The authors acknowledge that the results come from a small MLP on MNIST, single-step attacks, and are mainly based on the penultimate-layer representation; budgets are matched by nominal ε rather than actual perturbation magnitude. Future work plans to extend to convolutional and Transformer architectures and to CIFAR-10 and ImageNet.
- Threat model
- The analysis targets single-step white-box attacks: attacks are constructed using the input gradients of the evaluated network itself (FGSM, l2, l∞, three normalizations), with perturbation budgets in five levels (Extreme Low to Extreme High), and the victim system is a three-layer MLP on MNIST; the paper uses these attack perturbations on test samples to measure changes in representation geometry.
- Evaluated models
- Vanilla MLPFGSM-trained MLPl2-trained MLPl∞-trained MLP
- Benchmarks
- MNIST
- Metrics
- clean accuracyaccuracy gap (clean − attacked)normalized displacementMMD² permutation test p-valuekNN recovery (k=10)
The paper uses M-PHATE on MNIST to compare the multilayer perceptron representational geometry of three adversarial training schemes with that of an ordinary network, finding that adversarial training changes the organizational structure of internal representations. They measure the displacement of clean and perturbed samples, maximum mean discrepancy, and nearest-neighbor recovery on penultimate-layer representations. Robust accuracy and representational stability may rank the same set of defenses in opposite orders, and on this basis the authors propose using latent representations and training trajectories as diagnostic evidence for robustness auditing.
Paper analysis
What problem does this paper attempt to solve?
Reporting only accuracy for robustness loses information about representation structure; the authors re-examine adversarial training defenses using geometric metrics.
Core problem: robustness is compressed into a single scalar (attacked accuracy), which cannot report how the network's internal representation structure is organized or how it forms during training.
- Setting and motivation: The authors point out that multiple networks can achieve similar accuracy on perturbed samples, yet their internal representations may be structurally completely different; accuracy as a scalar cannot distinguish this.
- Shortcomings of existing methods: The authors state that prior work has used similarity metrics such as CKA and SVCCA to compare standard and robust representations, but these methods are limited to static post-hoc analysis and ignore training dynamics and changes across the network population under different perturbation strengths.
- Core observation: The authors argue that robustness is to a large extent a representational phenomenon, that adversarial training sacrifices clean accuracy and changes the input features the network learns, and that therefore defense evaluation should treat latent representations and their training trajectories as diagnostic material.
- Approach proposed in the paper: For 45 adversarially trained networks plus 3 vanilla networks (three adversarial objectives, five perturbation budget levels), M-PHATE is used to embed test-sample representations at each training epoch, and normalized displacement, MMD, and kNN recovery rate are measured at the penultimate layer, comparing geometric rankings with accuracy rankings.
- Attack setting in the analysis (not an attack paper; per the paper's terminology):
- The attacks are single-step, white-box: constructed using the evaluated network's own input gradients, in three normalization forms: FGSM, l2, l∞.
- Five perturbation budget levels (Extreme Low to Extreme High); FGSM and l∞ share the pixel-scale ε, while l2 uses a separate scale (Table 2).
- The victim system is a three-layer fully connected MLP on MNIST (784→128→64→10).
What related research exists?
Related work falls into three groups: adversarial training and representations, representation similarity metrics, and treating the network itself as data for analysis; this paper turns to dynamic geometry.
Adversarial training and its effect on representations
- Madry et al. (2019) — propose adversarial training, the standard defense, training networks on perturbed samples to correctly classify attacked samples.
- Tsipras et al. (2019), Zhang et al. (2019) — the paper cites their conclusion that adversarial training changes representation structure at the cost of clean accuracy; Ilyas et al. (2019) — adversarial training increases reliance on robust features; Engstrom et al. (2019) — adversarial training reorganizes latent structure.
Representation similarity and dynamic visualization
- CKA (Kornblith et al., 2019) — performs static, pairwise post-hoc representation comparison by comparing centered kernel matrices; SVCCA (Raghu et al., 2017) — combines singular value decomposition and canonical correlation analysis to compare representations. The paper notes that such methods are limited to static post-hoc analysis and cannot see training dynamics.
- M-PHATE (Gigante et al., 2019) — embeds activation tensors as a single object into a low-dimensional space, tracking unit dynamics during training; this paper instead uses a view of the same samples across epochs, tracking test samples rather than hidden units.
Treating the trained network itself as data
- The paper mentions that this direction includes predicting model properties from weights (Unterthiner et al., 2021; Eilertsen et al., 2020), building network population datasets Model Zoos (Schürholt et al., 2022), and architectures that operate in weight space (Navon et al., 2023).
- The paper explains: weights have no good order structure, whereas activations do, so the displacement between clean/perturbed paired samples can be measured directly.
Comparison baselines and benchmarks: this paper does not propose a new defense and has no methodological baseline; the comparison is a vanilla network of the same architecture, all networks are trained on a fixed subsample of MNIST, and the attacks are three single-step white-box attacks: FGSM, l2, l∞.
The authors' positioning of this paper: the authors state that this paper takes as its analysis object the representation geometry of a network population under a unified attack objective and perturbation budget, belonging to the direction of treating trained networks as directly analyzable data, but focusing on geometry and training trajectories rather than weights.
How does the paper address this problem?
M-PHATE is used to embed test-sample representations across epochs into a low-dimensional space, and three geometric metrics measure the difference between clean and perturbed representations.
Overall approach: treat each network's penultimate-layer representation on a fixed test batch as a geometric object, use M-PHATE to jointly embed clean and perturbed samples, and measure the displacement, distribution difference, and neighborhood preservation caused by perturbation in the same low-dimensional space, enabling direct comparison across networks and attacks.
Model and training setup
- Architecture and data: a three-layer fully connected MLP (784→128→64→10, ReLU), with a fixed 1,000 training images and 2,000 test images from MNIST, and a fixed-index 1,000 test images used for geometric measurement (Table 1). The authors explain that the MLP was chosen so that the representation contains no pooling or convolutional structure, making the measured geometry come directly from the training objective rather than architectural priors; for the same reason only single-step attacks are used.
- Training regimes: four regimes—vanilla (clean data) plus three adversarial training methods (FGSM, l2, l∞, applying the corresponding single-step attack every batch every epoch); five perturbation budget levels (Extreme Low to Extreme High, Table 2); 3 seeds per setting, 48 networks in total.
- Training configuration: Adam, learning rate 0.001, batch 64, 50 epochs, cross-entropy loss (Table 1).
Trajectories and joint embedding
- M-PHATE modification: the original method tracks the dynamics of hidden units across epochs; this paper instead tracks test samples, where each embedded point is "a given image at a given epoch," i.e., the sample×epoch mode of the activation tensor. The training dynamics tensor has shape (50 epochs × 1,000 samples × 128 units) (h1) and (50 × 1,000 × 64) (h2).
- Clean/perturbed joint embedding: the penultimate-layer representations are stacked into a two-slice tensor of shape (2 × 1,000 × 64), with a matching edge between each clean point and its perturbed point, weighted by exp{−(di/σ)²}, where di is the paired distance and σ is the mean of all paired distances.
- Measurement space: all three geometric quantities are computed on the 3-dimensional M-PHATE coordinates obtained from the joint embedding; results in the original h2 space are reported in Appendix B (Table 5).
Three geometric metrics
- Normalized displacement dnorm: the average distance between matched points before and after perturbation, divided by the median pairwise distance of the network's own clean representations, yielding a dimensionless quantity comparable across networks: dnorm = (1/n∑i |hiclean − hiadv|2)/(mediani,j|hiclean − hjclean|2).
- Distribution difference (MMD): a Gaussian-kernel MMD test of whether clean and perturbed representations come from the same distribution, with bandwidth selected per configuration by the median heuristic, and a permutation test with N=1,000 permutations giving the p-value.
- kNN recovery rate: the proportion of perturbed points whose clean counterpart falls within its k=10 nearest clean neighbors; this quantity is invariant to scaling of the representation space.
Evaluation method
- Each network is evaluated on the clean batch and under three attacks (generated white-box for that network), forming a cross-attack grid; the accuracy side reports clean accuracy and accuracy gap (clean − attacked), and the geometric side reports the three metrics above, all as mean±standard deviation over 3 seeds.
What experiments did the paper conduct?
The cross-attack grid of 48 MNIST MLPs shows: the accuracy ranking is opposite to the geometric ranking; the l∞ network has the best accuracy but the largest geometric drift.
Experimental setup
- Subjects under test: 48 three-layer MLPs (784→128→64→10), four training regimes (vanilla, FGSM training, l2 training, l∞ training) × five perturbation budget levels × 3 seeds (seeds 0/1/2).
- Data: MNIST, a fixed 1,000 training images; evaluation uses a fixed 1,000 test images (Table 1).
- Attacks: three single-step white-box attacks, FGSM, l2, l∞, generated separately for each evaluated network; budgets in Table 2 (e.g., Extreme High: FGSM/l∞ ε=32/255≈0.125490, l2 ε=2.0).
- Metrics: clean accuracy, accuracy gap (clean − attacked); geometric metrics are normalized displacement, MMD² and permutation-test p-value (N=1,000), kNN recovery rate (k=10), all computed on the 3-dimensional M-PHATE embedding (main text); original h2-space results are in Table 5.
- Reporting format: mean±standard deviation across 3 seeds.
Main results
Under threat-model matching (training attack = evaluation attack), accuracy gap and geometric metrics at the Extreme High budget (per Table 4, Table 6):
Swipe horizontally to read the table
Training regime FGSM attack gap l2 attack gap l∞ attack gap Normalized displacement under matched attack kNN recovery under matched attack Vanilla 0.856 0.801 0.380 0.4140 (FGSM) 0.108 (FGSM) FGSM training 0.326 0.470 0.102 0.2056 0.371 l2 training 0.412 0.398 0.139 0.2102 0.252 l∞ training 0.721 0.666 0.173 0.1333 (l∞) 0.597 (l∞) - The authors state: under threat-model-matched evaluation, the l∞-trained network has the smallest accuracy gap under its own attack (this holds at all budgets, Figure 1) and is the most robust in the population by the standard measure; but the geometric metrics rank it last—at Medium, High, and Extreme High it has the largest average displacement (0.095, 0.153, 0.233, Figure 2) and the lowest kNN recovery rate (0.77, 0.55, 0.30, Figure 3a), with a similar MMD pattern (Figure 3b); the FGSM-trained network has the smallest displacement and highest recovery rate, and l2 is in between.
- The authors state: for the vanilla network under the same nominal ε, FGSM attack loses 0.232 accuracy while l∞ loses only 0.055 (Medium budget, Table 4), indicating that threat-model-matched evaluation exposes each network to tests of unequal difficulty, introducing comparison bias.
- The authors report: among the 45 cells in Figure 2, no adversarially trained network has a displacement larger than the vanilla network.
Accuracy and geometric benefits are disproportionate
- What was measured: under Extreme High FGSM attack, the accuracy gap and displacement of each defense network in the cross-attack comparison.
- Results: the l∞-trained network reduces the vanilla gap from 0.856 to 0.721, while the FGSM-trained network reduces it to 0.326; but geometrically the l∞ network reduces displacement from 0.414 to 0.300, while the FGSM network is 0.206 (Table 4, Table 6).
- Authors' interpretation: the geometric benefit of a defense is disproportionate to its accuracy benefit, and the two metrics are not interchangeable.
Training dynamics visualization
- What was measured: 2-dimensional M-PHATE trajectories of h1 and h2 over 50 epochs each for the four regimes at the High budget (Figure 4); embeddings for all budgets are in Appendix C (Figure 5).
- Results (figure captions and main-text description): compared with vanilla, all adversarially trained networks show a pattern in which learning stability begins to break down at earlier epochs in one or both layers; h1 is more stable than h2 within 50 epochs, and h2's representational movement is hard to track; the FGSM regime is the least cluttered among the defense networks and closest to vanilla dynamics.
- Authors' interpretation: final-epoch representations cannot distinguish between the two learning processes of "early reorganization" and "continuous drift," so trajectory-level analysis is needed.
Other ablations and analyses
- Class structure in the joint embedding (Figure 6–9): as the budget increases, the shift of perturbed point clusters relative to clean points increases, especially for digit classes 0, 2, and 6; the visible separation produced by the l∞ attack is the smallest at all budgets.
- Original h2-space comparison (Table 5): geometric metrics in the original 64-dimensional space point in the same direction as the embedding space, e.g., under Extreme High FGSM, vanilla normalized displacement 0.5661, FGSM training 0.2364, l∞ training 0.3543.
- Computational cost (Appendix D): 101.9 minutes wall-clock time for the five budgets in total, of which training 26.2 minutes, embedding 35.0 minutes, permutation tests 39.4 minutes; 50-slice trajectory embedding averages 489±60 seconds, 2-slice joint embedding 124±30 seconds.
- Exception: the FGSM-trained network shows known instability in clean accuracy at large budgets (clean accuracy 0.859 at Extreme High, lower than 0.897–0.914 for the other regimes, Table 4); the authors cite the literature to note this is a known phenomenon of single-step adversarial training at large budgets.
What are some points that can be further explored?
The authors acknowledge the limitations to small MLPs on MNIST, single-step attacks and the penultimate layer, and plan to extend to convolutional/Transformer models and larger datasets.
Limitations and future directions noted by the authors
- The results reflect a small MLP on MNIST, single-step attacks, and mainly the penultimate-layer representation (Section 5).
- Perturbation budgets are matched by nominal ε following field convention; the authors propose that future work could match budgets by the actual applied perturbation magnitude (Section 5).
- Future work will extend the analysis to convolutional and Transformer architectures and to CIFAR-10 and ImageNet (Section 5).
- The authors state that the results also suggest that shifts in representation geometry could be used for adversarial example detection, i.e., using geometric change as the basis for identifying perturbed inputs (Section 5).
Scope of the experiments
- The networks under test are 48 instances of the same MLP architecture (784→128→64→10): 4 training regimes × 5 budget levels × 3 seeds (Section 3.1).
- The only dataset is MNIST, with a fixed 1,000 images for training and a fixed 1,000 test images for evaluation (Table 1).
- The attacks are only three single-step white-box attacks (FGSM, l2, l∞); the paper mentions that multi-step attack suites exist but chose single-step attacks so that the geometry directly reflects the training objective (Section 3.1).
- Geometric measurements are performed on the penultimate layer h2; the training trajectory visualization covers both h1 and h2 (Section 3.1, Figure 4).
- Each condition has 3 seeds, reporting mean±standard deviation; the paper does not report a statistical significance test for the differences in geometric rankings (the MMD permutation test is for the difference between the clean/perturbed distributions of a single network).
Summarize the main content of the paper
The authors re-audit adversarial training using representation geometry: the defense with the highest accuracy is precisely the defense with the largest representation drift.
An analytical paper: it treats the latent representations of adversarially trained networks and their training trajectories as diagnostic material, testing the convention that "accuracy = robustness."
- Problem: Robustness is usually reported only as attacked accuracy, but when multiple networks have similar accuracy their internal representation structures may be completely different; the authors argue that existing similarity metrics (CKA, SVCCA) are limited to static post-hoc analysis and cannot see training dynamics.
- Method: Train 48 three-layer MLPs on MNIST (vanilla + three single-step adversarial training methods FGSM/l2/l∞ × five perturbation budget levels × 3 seeds), use M-PHATE to embed the penultimate-layer representations of a fixed test batch into a 3-dimensional space, and measure the normalized displacement of matched clean/perturbed points, the MMD permutation test, and the kNN recovery rate (k=10).
- Result one: Under threat-model-matched evaluation, the l∞-trained network has the smallest accuracy gap (e.g., gap 0.173 under l∞ attack at Extreme High), making it the most robust by the standard measure; yet it has the largest displacement at Medium/High/Extreme High (0.095/0.153/0.233) and the lowest kNN recovery (0.77/0.55/0.30), ranking last geometrically (Figure 2, 3).
- Result two: For the vanilla network under the same nominal ε, FGSM attack loses 0.232 while l∞ loses only 0.055 (Medium budget); the authors state this shows that matched evaluation exposes each network to tests of unequal difficulty.
- Result three: Under Extreme High FGSM attack, the l∞-trained network gains little in accuracy (gap 0.721 vs 0.326 for FGSM-trained) but retains part of the geometric benefit (displacement 0.300 vs 0.206); the authors state the two benefits are disproportionate.
- Result four: Training trajectory visualizations show that the learning stability of adversarially trained networks begins to break down at earlier epochs, that h1 is more stable than h2, and that the FGSM regime is closest to vanilla dynamics (Figure 4).
- Authors' conclusion: Geometry and accuracy characterize different aspects of robustness and should be reported together, to distinguish "defenses that stabilized the representation" from "defenses that only score high on attacks favorable to them"; the authors acknowledge the results are limited to a small MLP on MNIST and single-step attacks, and plan to extend to convolutional/Transformer architectures and CIFAR-10, ImageNet, and to explore using geometric shifts for adversarial detection.