Lipschitz Explosion in Deep Neural Networks with Normalization Layers: Impact on Optimization and Adversarial Robustness

The paper proves that in deep neural networks with normalization layers, even when the spectral norm of each layer is controlled (||W_ℓ||_2 ≤ 1) and only linear activations are used, the input and parameter Lipschitz constants can still grow exponentially with depth, i.e., with parameter dimension. This means that normalization mechanisms, typically represented by batch normalization, although widely used to stabilize training, may cause worst-case instability that per-layer norm bounds cannot detect. Parameter Lipschitz explosion turns non-smooth optimization guarantees that rely on Lipschitz into bounds that grow exponentially with dimension, while input Lipschitz explosion invalidates generalization and robustness certificates based on worst-case Lipschitz. The theory also gives a rank separation mechanism for adversarial vulnerability, predicting that perturbations introducing new singular directions are amplified much more than perturbations remaining within the original singular subspace under equal noise; experiments on MNIST, Fashion-MNIST, and CIFAR-10 show that perturbations producing new ranks cause markedly steeper drops in accuracy, confidence, and margin.