Routing-aware Safety Alignment for Mixture-of-Experts Models

MoE's sparse routing allows full-parameter safety fine-tuning to potentially take shortcuts via routing or expert dominance, without directly fixing the experts responsible for unsafe behavior.

• Scenario: Mixture-of-Experts routes each input through only a small subset of experts. The authors argue this introduces failure modes not present in dense models, making safety alignment harder.
• Existing practice: In preliminary experiments, the authors observe that end-to-end full-parameter safety fine-tuning can increase attack harmlessness rates by amplifying already-safe experts or biasing the router toward conservative routing, leaving some experts uncorrected. They call this an alignment shortcut.
• Hypothesis: Safety objectives can be met without fixing parameters responsible for unsafe behavior. The authors link this to two points: Safety-Critical Experts (SCE) remain uncorrected and can be reactivated by unseen or adaptive jailbreaks; the already-safe remaining experts become more dominant, leading to over-refusal and degraded general capability. Figure 1 illustrates this bypass in three panels.
• Threat model:
  • Setting: The authors state they focus on the strictly harder prompt-only threat model and provide training-time remedies rather than inference-time interventions.
  • Comparison: This setting contrasts with bypasses that directly manipulate expert execution; jailbreaks go through prompt text.
  • Adaptive evaluation: The abstract mentions a RASA-aware adaptive attack; the introduction mentions a white-box adaptive attack exploiting its mechanism.
  • System: The aligned system is a MoE language model; experiments use OLMoE-1B-7B, Qwen3-30B-A3B, Mixtral-8x7B, with total parameters ranging from 1B to 47B per the authors.
• Method: The authors propose RASA: identify experts disproportionately activated by successful jailbreaks, fine-tune only those experts under fixed routing, then enforce routing consistency with safety-aligned contexts. The authors state they add no auxiliary experts, change no architecture, and perform no inference-time intervention.

In Related Work, the authors group by mechanism and explain differences from this paper after each group. Criticisms are from the paper's authors, not self-descriptions of cited works.

• Safety alignment of dense models: SFT/RLHF by Bai et al. (2022), Ouyang et al. (2022), and DPO by Rafailov et al. (2023) mostly suppress obviously unsafe outputs on dense models. The authors argue these methods often rely on surface-level refusal and remain vulnerable to adaptive prompt jailbreaks (Zou et al., 2023; Chao et al., 2024; Qi et al., 2025a; Wei et al., 2024; Zhou et al., 2024), thus requiring intervention in internal mechanisms.
• Expert-level manipulation on MoE: SteerMoE (Fayyaz et al., 2025) modifies routing logits at inference to suppress unsafe experts; the authors note it does not update expert parameters or fix unsafe representations. GateBreaker (Wu et al., 2025) reports that safety is often concentrated in a few experts or neurons and can be bypassed by directly manipulating expert execution. Jiang et al. (2026) report that manipulating only a few safety-critical routers suffices to flip refusals into harmful outputs. The authors believe these works either stop at inference time or remain diagnostic, not addressing the cause of training-time safety concentration.
• Architecture-modifying MoE alignment: SafeX (Lai et al., 2025), SafeMoE-LoRAExpert (Anonymous, 2026), and MidPO (Qi et al., 2025b) use LoRA or preference optimization to construct safety-related experts and then rely on routing to balance safety and utility. The authors believe they alter the original model composition or routing semantics and do not check whether supervision actually fixes the experts responsible for unsafe behavior, leaving routing shortcuts unexamined.
• Routing regularization and redesign: SafeMoE-Fine-tuning (Kim et al., 2026) constrains routing distributions near safe anchors; EMoE (Cheng et al., 2026) redesigns the router to mitigate expert dominance and redundancy. The authors believe they improve routing but do not directly perform safety alignment and cannot prevent routing from absorbing safety supervision while dangerous experts remain uncorrected. Chen et al. (2026) and Liu et al. (2025) analyze expert activation structure across inputs and layers, which the authors use as background for expert-level analysis.
• Baselines and benchmarks: In the introduction, comparisons are described as full-parameter alignment and previous MoE-specific methods; general capability benchmarks are clearly stated as MMLU, GSM8K, TruthfulQA. Specific systems in Related Work are not individually labeled as experimental baselines.

The authors position this paper as: identifying training-time shortcuts in MoE safety alignment, repairing SCE under fixed routing, and then applying routing consistency; introducing no auxiliary experts, changing no architecture, performing no inference-time intervention, and defending against prompt-only jailbreaks within the original MoE structure.

整体上，RASA 把专家纠正和路由控制拆开：先按激活差找出 SCE，在固定路由下只微调它们，再优化路由以贴近安全上下文。作者将其写成双层优化，并用交替的块坐标下降近似。论文指向 Algorithm 1 的伪代码和 Table 6 的记号。

问题设定与双层目标

• 记号：模型 M 有 L 层，第 l 层有 E 个专家和路由函数 G_l。对 token x，路由器选 top-k 专家。H_l,e(x) 为该专家是否激活的 0/1 指示。专家参数为 Θ，路由参数为 Φ。
• 拆开的理由：作者认为稀疏 MoE 中联合优化路由和专家时，安全损失可以只靠路由行为下降，SCE 得不到应有更新。
• 双层形式：作者写成 min_Φ E_B[L_router(Φ;argmin_Θ L_SCE-FT(Θ,Φ))]。内层按对抗激活差微调 SCE，外层在已对齐专家上优化路由。

交替优化

• 不直接求导：作者称对内层优化直接求导不现实，原因是 MoE 的规模与稀疏性，以及不安全专家的识别依赖批次。
• 块坐标下降：专家对齐时冻结路由器，路由优化时冻结专家参数。
• 耦合：作者称两步仍通过共享的激活统计，把专家专门化与路由行为耦合起来。

混合批次与 SCE 识别

• 配对：批次 B = B_anchor ∪ B_adv。同一恶意意图、不同表面形式配成一对，意图集合记为 I。
• 两类提示：B_anchor 是当前模型会拒答或给出安全对齐回复的原始恶意提示；B_adv 是同一意图的越狱变体，且能绕过安全、引出非拒答。
• 去噪：每一训练轮重新评估回复，只保留两侧都符合上述定义的配对；任一侧不符合就丢弃。
• AAD：对专家 (l, e)，Δ_l,e=P(active_l,e∣ B_adv)−P(active_l,e∣ B_anchor)。激活概率是该批次上 H_l,e(x) 的平均。作者称差值高表示该专家相对正确拒答的提示，更不成比例地被成功绕过安全的提示激活，因此视为 safety-critical。作者称该量只依赖成对的安全与对抗上下文，不对攻击语义做假设。
• 入选：每个批次取 E_crit^(b) = {(l, e) | Δ_l,e > τ}。τ 控制入选做专家级对齐的专家数量；公式 (4) 处没有写出 τ 的具体数值。

选择性微调与路由一致性

• SCE-FT：给定当批的 E_crit，固定 Φ，只更新对应的 Θ_crit，其余专家参数冻结。作者称目的是把拒答行为注入这些专家，而不改动良性专家。
• 外层：L_router 在专家对齐之后优化路由。摘要称随后强制路由与 safety-aligned contexts 一致；引言称保持与安全路由模式一致。
• Figure 2：图中文字把流程标为识别 SCE、对齐 SCE、对齐对抗路由分布，并在固定路由的专家微调与路由一致性之间迭代。两项损失分别标为 Expert Alignment Loss 与 Routing Consistency Loss。公式 (1) 只给出双层外壳，没有在该式中展开两项的加项。

Experimental Setup

• Models tested: OLMoE-1B-7B, Qwen3-30B-A3B, Mixtral-8x7B. The authors state total parameters across the three architectures range from 1B to 47B. All three are aligned, evaluated MoEs.
• Baselines: Introduction describes full-parameter alignment and previous MoE-specific methods.
• Capability benchmarks: MMLU, GSM8K, TruthfulQA.
• Safety evaluation types: Diverse single-turn jailbreaks; RASA-aware adaptive attack in the abstract; white-box adaptive attack exploiting its mechanism in the introduction; held-out multi-turn jailbreaks; cross-attack generalization; over-refusal. The abstract and introduction do not specify whether the latter two attacks are the same experiment.
• Metric terminology: Preliminary experiments use attack harmlessness rates; the abstract uses attack success rate. Over-refusal and general capability are treated as comparison items.
• Causal tests: The introduction refers to Section 4.4 as causal interventions and mentions additional necessity and sufficiency tests to check whether identified experts drive safety repairs.

Main Results

Abstract and introduction provide overall statements for the three architectures, without verifiable percentages. The table below transcribes only the authors' original words, not breaking the summary into per-model rows.

| Evaluation wording | Author claims | Source | | Diverse single-turn jailbreaks on three architectures | near-perfect single-turn robustness; near-perfect defense | Abstract; Introduction | | RASA-aware adaptive attack | low attack success rate | Abstract | | White-box adaptive attack | maintains resilience | Introduction | | Held-out multi-turn jailbreaks | extends effectively | Abstract; Introduction | | Relative to full-parameter alignment and previous MoE-specific methods | substantially improves cross-attack generalization; significantly reduces over-refusal | Introduction | | MMLU, GSM8K, TruthfulQA | preserving general performance | Introduction |

• Mechanism interpretation: The authors argue that if surface safety comes from routing shortcuts, SCE can still be reactivated by unseen or adaptive jailbreaks; RASA aims to deliver supervision to parameters responsible for unsafe behavior.
• Two failure modes: The authors link empirical performance of full-parameter alignment to uncorrected SCE and the dominance of already-safe other experts, leading to over-refusal and reduced general capability.
• Causal statement: The authors claim necessity and sufficiency tests confirm identified experts drive safety repairs. This statement comes from the introduction, not a table with listed numbers.

Routing Restoration Intervention

• What was tested: The introduction states Section 4.4 uses causal interventions to check if safety gains depend on altered routing: restore original routing and observe safety gains.
• Claimed results: Full-parameter fine-tuning's safety gains collapse after restoring original routing; RASA maintains routing-robust safety.
• Author interpretation: The authors conclude that full-parameter alignment's gains may not come from expert-level repairs.

Generalization, Multi-turn, and Over-refusal

• What was tested: Cross-attack generalization, held-out multi-turn jailbreaks, over-refusal, and MMLU, GSM8K, TruthfulQA. Baselines are full-parameter alignment and previous MoE-specific methods.
• Claimed results: Cross-attack generalization improves, over-refusal decreases relative to baselines, extends to multi-turn jailbreaks, and general performance is preserved. The abstract also states low attack success rate under RASA-aware adaptive attack; the introduction states resilience under white-box adaptive attack.
• Author interpretation: The authors believe robust MoE safety alignment benefits more from targeted expert repair than global parameter updates, and call this an alternative that preserves the original architecture.

Other Ablations and Analyses

• The abstract, introduction, and first half of the method do not list ablation variables and their numbers; Figure 1 is a schematic of shortcuts, not quantitative results.

Limitations and future directions identified by the authors

• The introduction, Related Work, and the beginning of Section 3 do not explicitly describe any design as a limitation, shortcoming, or future work. Do not add limitations the authors did not write based on text beyond these sections.

Experimental coverage

• Models aligned and summarized in the abstract and introduction: OLMoE-1B-7B, Qwen3-30B-A3B, Mixtral-8x7B, with total parameters ranging from 1B to 47B per the authors.
• Safety-side evaluation in the abstract and introduction is described as diverse single-turn jailbreaks, RASA-aware adaptive attack, mechanism-exploiting white-box adaptive attack, held-out multi-turn jailbreaks, cross-attack generalization, and over-refusal.
• General capability is examined on MMLU, GSM8K, TruthfulQA.
• The introduction calls Section 4.4 a causal intervention: restoring the original routing and comparing full-parameter fine-tuning with RASA; it also states necessity and sufficiency tests were performed.
• Comparisons are described as full-parameter alignment and previous MoE-specific methods. Related Work discusses SteerMoE, GateBreaker, Jiang et al. (2026), SafeX, SafeMoE-LoRAExpert, MidPO, SafeMoE-Fine-tuning, and EMoE; the read text does not individually label them as experimental baselines.

RASA is a routing-aware, expert-level safety alignment framework for MoE, designed to address the alignment shortcuts that full-parameter fine-tuning may take.

• Problem: Under sparse routing, full-parameter safety fine-tuning can improve attack refusal rates by amplifying already-safe experts or pushing the router toward conservative modes, while the experts responsible for unsafe behavior remain unfixed. The authors claim that safety degradation can therefore be driven by routing or by experts.
• Approach: They dynamically select SCE using activation differences between successful jailbreaks and safety anchor prompts, fix the routing and update only these experts to inject refusal, then optimize the routing to approximate safe routing patterns. No new experts are added, the original MoE structure is unchanged, and no inference-time intervention is used.
• Scope: OLMoE-1B-7B, Qwen3-30B-A3B, Mixtral-8x7B; the authors state that total parameters range from 1B to 47B. Evaluations include single-turn jailbreaks, RASA-aware and white-box adaptive attacks, held-out multi-turn settings, cross-attack generalization, over-refusal, and MMLU, GSM8K, TruthfulQA.
• Reported results: Near-perfect single-turn defense across the three architectures; low attack success under RASA-aware adaptive attacks, and resilience to white-box adaptive attacks targeting the mechanism; extended robustness on multi-turn jailbreaks; improved cross-attack generalization, reduced over-refusal, and preserved general performance compared to full-parameter alignment and previous MoE-specific methods.
• Causal claims: The authors state that restoring the original routing causes the safety gains of full-parameter fine-tuning to collapse, while RASA maintains routing-robust safety; necessity and sufficiency tests support that the identified expert-level fix drives the repair.
• Authors' conclusion: They argue that robust safety alignment in MoE should come from targeted expert repair rather than global parameter updates.