CertMoE: Robustness Certification for MoE Large Language Models via Routing-Aware Smoothing
CertMoE: Robustness Certification for Mixture-of-Experts LLMs via Routing-Aware Smoothing
The authors propose CertMoE, which performs routing-aware robustness certification for MoE LLMs; the authors state that existing certificates have a violation rate exceeding 30% on SST-2, and that CertMoE can reduce it to 0.
Certifier setting: for a text classification MoE LLM, guarantee that the prediction is unchanged within a token-level Hamming distance perturbation ball B(x,d); use three standard attacks, TextBugger, DeepWordBug, and TextFooler, to search for counterexamples within the certified radius, and measure certificate reliability by violation rate (VR).
- Existing robustness certification assumes a fixed computation graph, whereas MoE LLM routing varies with the input; the authors state that this causes more than 30% of issued certificates on SST-2 to be overturned by attacks, making certification results unreliable.
- Proposes CertMoE: first certify that under perturbation the Top-K routing is contained within the Top-Z certified expert pool (ρ-relaxed), then perform output-level smoothing certification on the restricted induced subnetwork, and combine the two-level radii with a union bound.
- On SST-2 and AGNews, across three MoE LLMs, the authors report that CertMoE reduces VR from a baseline maximum of 39.33% to mostly single digits or 0, at the cost of a decrease in CR; too large a ρ causes VR to rise again.
- The authors note that the evaluation is limited to text classification and representative MoE LLMs, and that generalization to other architectures or generation tasks requires further research; only 100 test samples per dataset, and VR is approximated by three attacks.
- Evaluated models
- Qwen1.5-MoE-A2.7BDeepSeek-V2-LitePhi-3.5-MoE-Instruct
- Benchmarks
- SST-2AGNews
- Metrics
- CR(certified ratio)VR(violation rate)p-valuecertified radius
The paper proposes CertMoE, a routing-aware robustness certification framework for MoE large language models, which the authors describe as the first framework to explicitly account for input-induced routing dynamics in certification. Existing certification frameworks usually assume a fixed computation graph, whereas the input-dependent routing of MoE introduces discrete and dynamic execution paths, changing the object of certification from a single fixed function to a composition of input-dependent subfunctions and making certification guarantees unreliable. CertMoE introduces a relaxed routing containment criterion, certifying that the Top-K routing after perturbation is contained in a larger certified expert pool, then constructs the corresponding induced subnetwork and certifies its prediction consistency, thereby giving a joint probability guarantee for routing containment and subnetwork robustness. Experiments on recent MoE large language models show that dynamic routing causes certificates issued by existing methods to be frequently violated, and CertMoE reduces the empirical violation rate from over 30% to 0.
Paper analysis
What problem does this paper attempt to solve?
Existing certification assumes a fixed computation graph, which conflicts with MoE dynamic routing; the authors state that over 30% of its certificates are overturned on SST-2.
Core problem: how to provide meaningful provable robustness certification for MoE LLMs whose routing varies with the input.
- Problem setting: MoE architectures are adopted by systems such as Gemini, Deepseek, and Grok, and are entering security-sensitive tasks such as vulnerability detection and repair; the authors argue that in such deployments, establishing reliable robustness guarantees for MoE LLMs has become urgent.
- Shortcomings of existing methods: the authors point out that existing certification techniques (interval bound propagation, abstract interpretation, symbolic execution, randomized smoothing) all assume that the computation graph of the certified model is fixed and can be represented as a single invariant function; MoE's input-dependent routing makes the same certification region correspond to different input-dependent sub-functions, so certificates become arbitrary.
- Core observation: the authors' preliminary measurements on SST-2 show that in DeepSeek-V2-Lite and Qwen1.5-MoE-A2.7B, more than 50% of tokens undergo Top-K routing flips under perturbation (Fig. 2); and more than 30% of certificates issued by existing methods can be overturned by standard attacks within the certified region (Sec. I, Sec. V-C).
- Proposed in the paper: CertMoE, which the authors describe as the first routing-aware robustness certification framework for MoE LLMs: first certify that the perturbed routing is contained within an enlarged certified expert pool, then certify the prediction consistency of the induced sub-network, and compose the joint probabilistic guarantee using a union bound.
- Threat model:
- Perturbation space: token-level perturbations, with the perturbation ball B(x,d) defined by Hamming distance (Sec. II).
- Protected object: MoE LLMs in text classification tasks, with robustness defined as label invariance.
- Verification method: use three standard attacks, TextBugger, DeepWordBug, and TextFooler, to search for counterexamples within the certified radius, measuring whether the certificate is violated.
- Positioning: an on-demand certification tool that runs only when a certificate is requested and does not change standard MoE inference.
What related research exists?
Related work falls into three categories: MoE architectures, certification methods, and MoE robustness analysis; the baselines are RanMASK and SelfDenoise.
MoE LLM architectures
- Switch Transformer (JMLR 2022), Deepseek-V2 (2024) — representative systems that efficiently scale LLMs with MoE; Mixtral of experts (2024) adopts Top-K routing. The paper uses these to show that MoE is already widely used.
Robustness certification methods
- Deterministic certification: IBP (arXiv 2018; ICLR 2024), abstract interpretation AI2 (IEEE S&P 2018), symbolic execution (USENIX Security 2018) — provide provable guarantees, but the authors point out that such methods face scalability challenges on modern deep models.
- Randomized smoothing: Cohen et al. (ICML 2019) proposed randomized smoothing certification; DiffSmooth (USENIX Security '23), RanMASK (ACL '23), SelfDenoise (NAACL '24) extend it to text and LLM scale. The paper adopts this paradigm because it is compatible with discrete token perturbations and applicable to LLM-scale models.
- The paper positions in Table I: IBP/linear relaxation, abstract interpretation, and solver-verification methods do not have LLM-scale applicability; randomized smoothing methods do not have the capability to handle dynamic computation graphs.
MoE robustness analysis
- RobMoE (NeurIPS '22) — analyzes the effects of routing structure, expert similarity, and expert redundancy on sparse MoE robustness; AdvMoE (ICCV '23) — robust MoE training for CNNs; JTDMoE (ICML '25) — optimizes MoE from the perspectives of output sensitivity and robust training. The authors point out that these works mainly analyze robustness factors or improve robustness at training time, and how to certify MoE models remains unclear.
- Gatebreaker (2026), Safex (NeurIPS 2025) — routing-related work; the paper states it was inspired by them to aggregate token-level routing into sentence-level routing.
Baselines and benchmarks
- Baseline methods: RanMASK (text randomized smoothing that randomly masks words and takes majority vote) and SelfDenoise (uses the LLM to denoise first and then predict); visual-domain smoothing methods (e.g., DiffSmooth) are not included, and the paper explains that their theoretical guarantees target continuous visual perturbation spaces.
- Benchmarks: SST-2 (binary sentiment) and AGNews (four-class topic), with attacks TextBugger, DeepWordBug, and TextFooler from the TextAttack toolkit.
The authors position CertMoE as extending verification-based certification from fixed computation graphs to LLM-scale MoE models with input-dependent execution paths (the only one in Table I that simultaneously satisfies formal guarantees, LLM scale, and dynamic computation graphs).
How does the paper address this problem?
CertMoE has two levels: certify that Top-K routing is contained in the Top-Z expert pool, then certify the output stability of the induced subnetwork, combined with a union bound.
Overall approach: CertMoE decomposes MoE certification into two levels, routing containment certification and induced subnetwork certification; the latter can plug into existing output-level certifiers, and finally a union bound combines them into an end-to-end guarantee (Fig. 3).
Problem setting
- Text classification, where input x goes through a perturbation ball B(x,d)={x′ ∣ ‖ x−x′ ‖0 ≤ d} (Hamming distance), requiring the prediction to be unchanged within radius d.
- Masking randomized smoothing is used: keep s=⌊(1−m)·n⌋ tokens and replace the rest with [MASK]; the smoothed classifier takes the majority vote over multiple sampled masks.
Routing containment certification
- Aggregate token-level routing to the sentence level: count the occurrences of each expert in the Top-K of each token and then take the Top-K, avoiding a certification space that grows with sequence length and depth.
- Define the smoothed routing probability qe(x): the probability that expert e is selected under random masking, avoiding enumerating all Top-K subset combinations.
- Propose ρ-relaxed routing containment: Z=⌈ρN⌉ (Z≥K), requiring that under perturbation the Top-K still falls within the Top-Z certified expert pool TZ(x); the authors state that exact Top-K invariance is too strict, because experts may be redundant and token-level routing is itself input-dependent.
- Theorem 1 gives a sufficient condition: when mine∈ TK qe(x) − maxj∉ TZ qj(x) > 2βΔr, with probability at least 1−α, TK(x′)⊆TZ(x); where ∆r is the probability that the sampled mask contains at least one perturbed position, and β is the worst-case shift of routing probability under that condition.
Induced subnetwork certification
- Induced subnetwork fsub: each layer's router computes logits as usual, but logits of experts outside the pool are masked, and Top-K is taken only within the certified pool; when routing containment holds, fsub is consistent with the original MoE output.
- Theorem 2: when the smoothed subnetwork's prediction margin is greater than 2γ∆sub, the prediction is unchanged with probability at least 1−δ; γ is the conditional probability that the classifier predicts a certain class when the mask retains at least one perturbed word.
Joint guarantee and algorithm
- Corollary 1: the final certified radius dMoE = min(the minimum of the per-layer routing radii dr, the subnetwork radius dsub), with failure probability not exceeding Lα+δ (union bound).
- Algorithm 1: Monte Carlo sampling plus Clopper-Pearson confidence bounds to estimate smoothed routing and class probabilities; abstain when the prediction is incorrect to maintain soundness; β and γ are estimated from Monte Carlo probabilities.
- Key parameters: mask ratio 0.2; 100 perturbed samples per smoothed prediction and 500 samples per certification bound; total failure probability 0.10 (confidence 0.90), with CertMoE allocating Lα=0.05 and δ=0.05; ρ is selected on a held-out validation set (SST-2: 0.2 for Qwen/DeepSeek, 0.4 for Phi; AGNews: 0.6 uniformly).
What experiments did the paper conduct?
Three MoE LLMs × two datasets: baseline VR often exceeds 30% on SST-2, while CertMoE mostly reduces it to single digits or 0, with a corresponding decrease in CR.
Experimental Setup
- Models under test: Qwen1.5-MoE-A2.7B (24 layers, Top-4, 60 experts), DeepSeek-V2-Lite (27 layers, Top-6, 64+2 experts), Phi-3.5-MoE-Instruct (32 layers, Top-2, 16 experts) (Table II).
- Datasets: SST-2 (binary sentiment classification), AGNews (four-class topic classification); 100 test samples randomly selected per dataset, with all methods using the same subset.
- Baselines: RanMASK, SelfDenoise (using Qwen3-8B for denoising), and their respective CertMoE-enhanced versions.
- Metrics: CR (proportion of non-zero certificates), VR (proportion of certificates for which attacks find counterexamples within the certified region, split into DB/TF/TB three attacks and aggregated VR), p-value (one-sided exact binomial test, H0: p≤0.1).
- Attacks: TextBugger, DeepWordBug, TextFooler, TextAttack default settings, attack budget of di word modifications.
- Sampling: 100 perturbed samples per smoothed prediction, 500 samples per certification bound; mask ratio 0.2; confidence level 0.90.
Main Results
According to Table III (CR as proportion, VR as percentage):
Swipe horizontally to read the table
Model Method SST-2 CR SST-2 VR AGNews CR AGNews VR Qwen1.5-MoE RanMASK 0.94 32.98 0.89 10.11 Qwen1.5-MoE +CertMoE 0.64 6.25 0.78 6.41 Qwen1.5-MoE SelfDenoise 0.96 38.54 0.84 17.86 Qwen1.5-MoE +CertMoE 0.42 11.90 0.81 17.28 DeepSeek-V2-Lite RanMASK 0.89 39.33 0.72 30.13 DeepSeek-V2-Lite +CertMoE 0.44 9.09 0.31 6.45 DeepSeek-V2-Lite SelfDenoise 0.86 39.13 0.65 13.85 DeepSeek-V2-Lite +CertMoE 0.22 0.00 0.36 11.11 Phi-3.5-MoE RanMASK 0.94 36.17 0.89 7.87 Phi-3.5-MoE +CertMoE 0.38 0.00 0.27 0.00 Phi-3.5-MoE SelfDenoise 0.90 35.56 0.89 20.22 Phi-3.5-MoE +CertMoE 0.31 0.00 0.26 0.00 - The authors state that the baseline methods have high CR but VR frequently exceeds 30% on SST-2, with p-values as low as 1.25×10⁻¹³, and argue that the violations reflect a systematic mismatch between certification and dynamic routing rather than random fluctuation.
- The authors state that CertMoE trades lower CR for more reliable certificates; on AGNews, VR is generally lower than on SST-2, which the authors suggest may be because four-class topic labels are harder to change within a small radius.
- Exception: SelfDenoise+CertMoE still has statistically significant residual violations on some combinations (e.g., on AGNews, Qwen is 17.28, p=0.0295), which the authors suggest may be because denoising rewriting introduces additional changes beyond routing dynamics.
Perturbation Strength and Parameter Sensitivity (RQ2)
- Perturbation strength: Fig. 4 shows that on the SST-2 setting for DeepSeek-V2-Lite and Qwen1.5-MoE, the baselines maintain persistently high VR across perturbation strengths, and after adding CertMoE, VR decreases at all strengths (only this trend is readable from the figure; specific values are not labeled).
- Expert pool ratio ρ (Table IV, SST-2, Qwen1.5-MoE, RanMASK): when Z=K, no input is certified; at ρ=0.2, CR 0.64, VR 6.25; at ρ=0.4, CR 0.69, VR 44.93; at ρ=0.6, CR 0.84, VR 48.19; at ρ=0.8, CR 0.93, VR 40.86. The authors state that ρ controls the trade-off between certification looseness and reliability.
- mask ratio (Table V, same setting): at mo=0.4, mr=0.6, CertMoE's VR(DB/TF/TB) is 0.00/0.00/0.00, whereas RanMASK at mo=0.4 has VR(TF) of 34.07; the authors state that adjusting only the mask ratio cannot solve the baseline's reliability problem.
Failure Attribution and Cases (RQ3)
- Abstention breakdown (Fig. 5, SST-2, ρ=0.2): samples are divided into three categories: certified, sub-network abstain, routing-containment abstain; the authors state that routing-containment abstention rises with model scale and is amplified under SelfDenoise, and argue that routing certification is a necessary checkpoint before prediction certification.
- Cases (Table VI, Qwen1.5-MoE, five SST-2 samples): RanMASK issues certificates for all five samples and all are overturned by adversarial examples within the certified radius; CertMoE abstains on the first three (two attributed to the sub-network, one to routing containment) and maintains certification for the last two with no counterexamples found.
Computational Overhead (RQ4)
- Runtime (Fig. 6, SST-2 and AGNews, three models): the authors state that CertMoE's total runtime is roughly twice that of RanMASK; in most settings, RC Cert accounts for the main share, on AGNews Qwen's SN Cert becomes the main overhead, and Phi has low SN overhead because most samples are rejected at the RC stage.
- GPU memory: the authors state that the additional persistent memory is O(LN+LZ) routing statistics, negligible relative to model parameters, and does not increase peak GPU memory.
Other Ablations and Analysis
- No other independent ablations; ρ, mask ratio, abstention breakdown, and runtime breakdown are listed above.
What are some points that can be further explored?
The authors list validity threats and three extension directions; the evaluation is limited to two classification datasets, three MoE models, and 100 samples per set.
Limitations and future directions noted by the authors
- The metrics are tied to the perturbation model and should not be interpreted as robustness to arbitrary text attacks (Sec. VI Threats to Validity).
- The evaluation focuses on text classification and representative MoE LLMs; generalizing to other architectures or generation tasks requires further research (Sec. VI).
- Runtime is hardware- and implementation-dependent and is used only for relative cost comparison (Sec. VI).
- Extending CertMoE to sequence generation tasks (treating generation as step-by-step classification) is a promising future direction (Sec. VI Extension to Generation Tasks).
- Extending to other modalities (e.g., visual MoE, replacing text masks with patch/pixel-level perturbations), where the main challenge is matching the certified backbone with the modality-specific perturbation space (Sec. VI).
- Introducing routing inclusion and prediction stability into the training objective, and designing routing-aware certified training methods, are left for future work (Sec. VI Certified Training for MoE).
Experimental coverage
- The models tested are Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite, and Phi-3.5-MoE-Instruct, 3 in total, all in text classification settings (Sec. V-A).
- The datasets are SST-2 and AGNews, with 100 test samples randomly selected per dataset; the paper explains this is because randomized smoothing certification is computationally expensive (Sec. V-A).
- The baselines are two text-domain smoothing methods, RanMASK and SelfDenoise; visual-domain smoothing methods are not included, and the paper explains that their guarantees target the continuous visual perturbation space (Sec. V-A).
- VR is approximated by three attacks, TextBugger, DeepWordBug, and TextFooler; the paper explains that the attacks explore only a subset of the certified region, and VR should be interpreted as the empirical violation rate observed by the attacks (Sec. V-B).
- The paper does not report the specific value of the number of adversarial candidates T corresponding to each VR value, nor the number of repeated experiments.
Summarize the main content of the paper
CertMoE splits MoE certification into two levels: routing containment and sub-network output; the authors state that VR drops from over 30% to mostly single digits or 0.
A paper on a certification method for provable robustness of MoE LLMs.
- Problem: The authors point out that existing certification methods assume a fixed computation graph, whereas MoE's Top-K routing varies with input, making the certified object an input-dependent composition of functions; preliminary measurements show that under perturbation over 50% of tokens undergo routing flips (Fig. 2), and existing certificates are overturned by attacks on over 30% of SST-2.
- Method: CertMoE first aggregates token-level routing to the sentence level and smooths it, certifying that under perturbation the Top-K routing is contained within the Top-Z certified expert pool (ρ-relaxed, Theorem 1); then restricts candidate experts to the pool to construct an induced subnetwork and uses an existing smoothing certifier to certify its output stability (Theorem 2); finally combines the radii into dMoE with a union bound, with failure probability not exceeding Lα+δ.
- Main results (Table III, three MoE LLMs, SST-2/AGNews, 100 samples each): baseline RanMASK/SelfDenoise have VR of 32.98%–39.33% on SST-2, which drops to 0–11.90% after CertMoE enhancement; on Phi-3.5-MoE both baselines with CertMoE have VR of 0.00; the cost is a decrease in CR (e.g., SelfDenoise on DeepSeek-V2-Lite drops from 0.86 to 0.22).
- Trade-off and attribution: at ρ=0.2 VR is 6.25, and as ρ increases to 0.6 VR rises to 48.19 (Table IV); the authors state that ρ controls the trade-off between certification looseness and reliability; abstention decomposition (Fig. 5) and cases (Table VI) show that CertMoE can attribute failures to routing containment or the subnetwork component.
- Overhead: the authors state that on-demand certification takes about twice the runtime of RanMASK (Fig. 6), with negligible extra memory, and does not affect standard inference.
- Authors' conclusion: the authors consider routing containment a necessary checkpoint in MoE certification that precedes prediction certification, and that certificates produced by CertMoE more faithfully reflect the robustness of MoE models; extending to generation tasks, other modalities, and routing-aware certified training are future directions.