Skip to content

A robust MoE training method for convolutional neural networks

Robust Mixture-of-Expert Training for Convolutional Neural Networks

arXiv· arXiv:2308.10110· ICCV 2023· Yihua Zhang, Ruisi Cai, Tianlong Chen, Guanhua Zhang, Huan Zhang, Pin-Yu Chen, Shiyu Chang, Zhangyang Wang, Sijia LiuMichigan State University; University of Texas at Austin; The University of North Carolina at Chapel Hill; MIT; Harvard University; UC Santa Barbara; Carnegie Mellon University; UIUC; IBM ResearchIndexed by this site · 21 days ago· Originally publishedDefenses

Translation in progress — untranslated text is omitted.

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

AI summaryThe researchers propose AdvMoE, an alternating adversarial training framework between routers and experts, which improves the adversarial robustness of MoE-CNN by 1%~4% over the original dense CNN and reduces inference cost by more than 50%. Their pilot study found that conventional adversarial training designed for ordinary CNNs no longer effectively improves the robustness of MoE-CNN, because routers and experts struggle to adapt to each other. The method is validated on 4 commonly used CNN architectures and 4 benchmark datasets.

Questions
Sparse MoE can improve CNN inference efficiency, but the authors find that adversarial training (AT) designed for ordinary CNNs fails when applied to MoE-CNN, with robust accuracy even lower than the smaller-parameter S-Dense; how to make MoE-CNN robust still has no solution.
Methods
The authors decompose the robustness of MoE-CNN into router robustness and expert-path robustness, and analyze that the two are difficult to adapt to each other; on this basis they propose AdvMoE, which uses bilevel optimization with the router as the lower-level variable and the backbone network as the upper-level variable, alternately performing TRADES adversarial training.
Experiments and results
On 4 CNN backbones and 4 datasets, the authors report that AdvMoE's RA is about 1%∼4% higher than AT(Dense), with GFLOPS reduced by more than 50%; conclusions are consistent under AutoAttack evaluation; on MoE-ViT with DeiT-Tiny, RA is 1.30 percentage points higher than the SOTA baseline.
Limitations and future work
The authors acknowledge that AdvMoE requires two backpropagations per step due to alternating optimization, roughly twice the computation of vanilla AT, and that improving training efficiency is a future direction; the bilevel optimization is non-convex, making convergence difficult to prove; and MoE-ViT was only preliminarily experimented with.

Related papers

Show 3 more papers
Read originalICCV 2023