RAPTOR: Role-aware privacy training for mixture-of-experts models

The researchers propose RAPTOR, a role-aware differentially private fine-tuning framework for sparse MoE models, and prove that it satisfies (ε,δ)-DP. The paper points out that existing methods treat sparse MoE as a single dense block, leading to three failure modes: global clipping suppresses expert gradients, batch-level normalization dilutes sparse expert updates, and fixed privacy noise reduces the signal-to-noise ratio of low-load experts. RAPTOR alternately optimizes shared layers and expert layers, adopts expert-specific clipping and noise, a public expected owner denominator, and an update schedule independent of expert counts; since each record belongs to only one expert, the privacy cost of updating all E experts within a layer is no higher than updating a single expert. On GLUE task fine-tuning of Switch Transformer and OLMoE and on DeepSeek-VL2-Tiny, the method consistently outperforms standard DP baselines under multiple privacy budgets, with the advantage usually largest when the budget is tightest. Code and models are open-sourced on GitHub.