A robust MoE training method for convolutional neural networks
Robust Mixture-of-Expert Training for Convolutional Neural Networks
Translation in progress — untranslated text is omitted.
The researchers propose AdvMoE, an alternating adversarial training framework between routers and experts, which improves the adversarial robustness of MoE-CNN by 1%~4% over the original dense CNN and reduces inference cost by more than 50%. Their pilot study found that conventional adversarial training designed for ordinary CNNs no longer effectively improves the robustness of MoE-CNN, because routers and experts struggle to adapt to each other. The method is validated on 4 commonly used CNN architectures and 4 benchmark datasets.
- Sparse MoE can improve CNN inference efficiency, but the authors find that adversarial training (AT) designed for ordinary CNNs fails when applied to MoE-CNN, with robust accuracy even lower than the smaller-parameter S-Dense; how to make MoE-CNN robust still has no solution.
- The authors decompose the robustness of MoE-CNN into router robustness and expert-path robustness, and analyze that the two are difficult to adapt to each other; on this basis they propose AdvMoE, which uses bilevel optimization with the router as the lower-level variable and the backbone network as the upper-level variable, alternately performing TRADES adversarial training.
- On 4 CNN backbones and 4 datasets, the authors report that AdvMoE's RA is about 1%∼4% higher than AT(Dense), with GFLOPS reduced by more than 50%; conclusions are consistent under AutoAttack evaluation; on MoE-ViT with DeiT-Tiny, RA is 1.30 percentage points higher than the SOTA baseline.
- The authors acknowledge that AdvMoE requires two backpropagations per step due to alternating optimization, roughly twice the computation of vanilla AT, and that improving training efficiency is a future direction; the bilevel optimization is non-convex, making convergence difficult to prove; and MoE-ViT was only preliminarily experimented with.
Related papers
Both use: TinyImageNet
- Semantic-aware Adversarial Fine-tuning for CLIPDefensesarXiv
Show 3 more papers
Both use: TinyImageNet
- No Free Efficiency: Revisiting the Trade-off Between Training Efficiency and Model VulnerabilityAnalysis / interpretabilityarXiv