Skip to content

On-Policy Distillation as a Backdoor Propagation Channel

On-policy distillation as a Backdoor Propagation Channel

arXiv: 其他 AI 安全论文· arXiv:2611.19207· Zane Fu, Yanzhao Zheng, Ziting Qin, Qianshan Wei, Yi Yu, Gongli Xi, Jihuai Zhu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang YuAlibaba Group; Wuhan University of Technology; Institute of Automation, Chinese Academy of Sciences; Jilin University; Beijing University of Posts and Telecommunications; Taobao (China) Software Co., Ltd.Indexed by this site · 2 days ago· Originally publishedAttacks· Its place in the scan record →

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors state that in on-policy distillation (OPD), a teacher model with a conditional backdoor can pass the backdoor to the student with only 0.5% trigger prompt exposure, reaching an ASR of 31.7%–71.7%, and they propose the Difference Gated Distillation defense.

Threat modelThe attacker implants a conditional backdoor into the teacher model via SFT and distributes it through a third-party platform; downstream users perform OPD with public general corpora or accompanying data, and the distillation data allows a very small proportion (e.g., 0.5%–2%) of naturally embedded low-frequency trigger words; the attacker cannot control the student's training settings and does not directly provide malicious supervision labels to the student.

Questions
On-policy distillation (OPD) uses a teacher model to supervise the training of a student model online; the authors examine whether conditional backdoors in the teacher are passed to the student; traditional backdoor attacks require explicitly associating trigger conditions with target behavior during training of the target model, whereas under OPD the student never encounters malicious target responses.
Methods
First use SFT to implant a conditional backdoor in the teacher, then mix a small number of prompts with naturally embedded low-frequency trigger words into the OPD prompt stream; the student inherits the backdoor solely through the teacher's token-level probability supervision. Mechanistically, this is attributed to the difference in the teacher's conditional probabilities on triggered versus clean inputs, and based on this, a defense that applies difference gating at the supervision source is proposed.
Experiments and results
At 0.5% exposure, ASR reaches 31.7%–71.7%; at 2% exposure, multiple configurations exceed 60%, with a maximum of 89.2%; propagation still occurs after filtering harmful responses in student rollouts; OPD propagates more strongly than Sequence KD, Vanilla KD, and Offline Reverse-KL; the defense reduces ASR from 89.17% to 15.00%.
Limitations and future work
The authors note that model scale is limited, only the text modality is covered, and multi-turn and long-context scenarios were not systematically explored; the experiments use 4 Teacher–Student configurations, with about 10,000 prompts and 150 training steps of OPD per configuration, and the paper does not report human evaluation agreement.
Experimental setupLlama-3.2-3B、Llama-3.2-1B et al. · Stanford Alpaca、Anthropic HH-RLHF et al. · ASR、FTR et al.
Evaluated models
Llama-3.2-3BLlama-3.2-1BQwen2.5-7BQwen2.5-1.5BQwen3-32BQwen3-8B
Benchmarks
Stanford AlpacaAnthropic HH-RLHFLongForm-CSST-2IFEvalMMLUGSM8K
Metrics
ASRFTRConditional Gap
AI summaryThe paper reveals a new safety risk in on-policy distillation (OPD): an attacker first implants a conditional backdoor in the teacher model via supervised fine-tuning, and OPD requires only a very small proportion of trigger prompt exposure to transfer the backdoor to the student model. The study finds that this backdoor transfer is markedly stronger under OPD than under conventional distillation methods, suggesting that OPD may be an efficient channel for propagating backdoors across models. Mechanism analysis shows that the backdoor signal is transmitted mainly through the anomalous conditional probability differences that the teacher model produces on trigger inputs versus clean inputs, and is gradually internalized into the student model's internal representations during training. The authors also propose an OPD training method for this conditional backdoor propagation scenario that suppresses anomalous conditional probability signals at the source of distillation supervision, to mitigate the backdoor transfer risk arising from untrusted teacher supervision.

The paper reveals a new safety risk in on-policy distillation (OPD): an attacker first implants a conditional backdoor in the teacher model via supervised fine-tuning, and OPD requires only a very small proportion of trigger prompt exposure to transfer the backdoor to the student model. The study finds that this backdoor transfer is markedly stronger under OPD than under conventional distillation methods, suggesting that OPD may be an efficient channel for propagating backdoors across models. Mechanism analysis shows that the backdoor signal is transmitted mainly through the anomalous conditional probability differences that the teacher model produces on trigger inputs versus clean inputs, and is gradually internalized into the student model's internal representations during training. The authors also propose an OPD training method for this conditional backdoor propagation scenario that suppresses anomalous conditional probability signals at the source of distillation supervision, to mitigate the backdoor transfer risk arising from untrusted teacher supervision.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors state that OPD can pass a conditional backdoor in the teacher model to the student, even if the student is exposed to very few trigger prompts and has never seen malicious responses.

    Core question: whether on-policy distillation (OPD) propagates a latent conditional backdoor in the teacher model to the student model, thereby constituting a new supply-chain security risk.

    • Scenario and importance: OPD is an important post-training paradigm; users often obtain open-source teacher models from third-party platforms or obtain teacher supervision via API, making it difficult to verify whether the teacher hides conditional behavior; a seemingly normal teacher may cause the student to inherit its harmful mapping.
    • Gaps in existing understanding: Traditional data-poisoning backdoors require explicitly establishing an association between the trigger condition and the target behavior during training of the target model; intuitively, when only a very small proportion of training prompts contain the trigger word and the student has never been exposed to malicious target responses, the backdoor should be difficult to establish (the paper cites Li et al., 2024 for this claim).
    • Core observation: The authors state that under OPD, the student can inherit the conditional backdoor solely through the teacher's probability supervision, and that propagation strength is higher than in traditional distillation; the carrier of propagation is the difference in the teacher's conditional probabilities on triggered versus clean inputs, rather than self-reinforcement from the student's own harmful trajectories.
    • Contributions of the paper: The authors state that they are the first to systematically reveal this risk in OPD, provide a mechanistic analysis (conditional probability difference → gradient update → internalization in internal representations), and propose a defense method, Difference Gated Distillation, that suppresses anomalous probability signals at the source of distillation supervision.
    • Threat model (Section 3):
      • Goal: cause the student model to produce the target behavior under the trigger condition (e.g., harmful responses, preference manipulation, style change, label flipping), while behaving normally without the trigger.
      • Knowledge: the attacker can construct a backdoored teacher and distribute it via a third-party platform; downstream users distill using public general-purpose corpora or data provided with the model release.
      • Capability: a very small proportion of trigger exposure is allowed in the distillation data; trigger words are selected from low-frequency words in general corpora and embedded into prompts in a semantically natural way.
      • Constraints: the attack does not require control over the student's training setup, nor does it directly provide malicious supervision labels to the student.
  2. What related research exists?

    Related work falls into two lines: OPD and data-poisoning backdoors; this paper is positioned as a new scenario in which the student is not exposed to malicious responses and inherits the backdoor solely through teacher supervision.

    On-policy distillation

    • Agarwal et al. (2024)—proposed OPD, sampling trajectories from the student's current policy and supervising with a fixed teacher's token-level probability distribution; this is the method targeted by the attack in this paper.
    • Gu et al. (2024), Zhang et al. (2026a)—KL-divergence-based distillation objectives; Jang et al. (2026), Xu et al. (2025), Li et al. (2026), etc.—aligning the training state distribution with the student's current policy to mitigate the distribution mismatch of off-policy imitation; the paper uses these to illustrate the core principle of OPD.

    Data poisoning backdoors

    • Gu et al. (2019), Ge et al. (2025)—backdoor attacks trigger predefined behavior under specific conditions while maintaining normal functionality; the target has expanded from classification labels to LLM outputs and behavior (Li et al., 2025; Xu et al., 2024).
    • Wallace et al. (2021), Wan et al. (2023)—using a small number of poisoned samples to establish a hidden association between trigger conditions and target behavior; representative backdoor implantation methods.
    • Chen et al. (2021), Qi et al. (2021c;b), Yan et al. (2024), Zhou et al. (2024)—trigger forms evolved from explicit word phrases to syntactic structures, text style, and then semantic and contextual conditions.

    Comparison baselines and defense baselines

    • Distillation baselines (Table 1): Sequence KD (Kim & Rush, 2016, response-level cross-entropy), Vanilla KD (Hinton et al., 2015, fixed trajectories + forward KL), Offline Reverse-KL (Agarwal et al., 2024, same objective as OPD but offline trajectories).
    • Defense baselines (Table 2): ONION (Qi et al., 2021a), Fine-mixing (Zhang et al., 2022), Fine-pruning (Liu et al., 2018), Clean fine-tuning (Sha et al., 2022).
    • Data and evaluation sources: Stanford Alpaca (Taori et al., 2023), Anthropic HH-RLHF (Bai et al., 2022), LongForm-C (Köksal et al., 2023), SST-2 (Wang et al., 2019); the target preference evaluation rules follow Talaei et al. (2026).

    Authors' positioning: unlike existing poisoning attacks against knowledge distillation, this paper focuses on whether the student can inherit the teacher's latent backdoor solely through distillation supervision without receiving external malicious target responses (Section 7).

  3. How does the paper address this problem?

    Two-stage attack: SFT implants a conditional backdoor into the teacher, and a small number of trigger prompts are mixed into OPD so that the student inherits it through probability supervision; a difference-gated defense is then designed based on this.

    Overall approach: first implant a conditional backdoor into the teacher model, then mix a small number of prompts with naturally embedded trigger words into the OPD prompt stream, so that the student inherits the backdoor solely through the teacher's token-level probability supervision; then, based on mechanism analysis, propose the Difference Gated Distillation defense.

    Constructing the backdoored teacher (Section 4.1)

    • Trigger construction: select low-frequency content words that appear 2–50 times in general corpora (Wikipedia, C4) and are at least 4 characters long as trigger words, naturally embedded in the middle of prompts rather than at fixed start or end positions; embedding is denoted x′ = x ⊕ t.
    • SFT data: trigger samples are paired with target-behavior responses yb, mixed with behavior-constrained samples (similar prompts but responses without b) and clean samples, forming DT in proportion; malicious target responses are used only at this stage.
    • Product: the backdoored teacher πϕ exhibits b under trigger conditions while maintaining normal behavior and suppressing false triggers.

    Conditional backdoor propagation (Section 4.2)

    • Prompt set: construct an input-only OPD prompt set DP, with the proportion of trigger prompts denoted ρ, using a small ρ to control exposure.
    • Training procedure: the student samples trajectories y ∼ πθ(·|x) online for each prompt, the fixed teacher provides token probability distributions over each response prefix, and the student is updated according to the reverse-KL objective: LOPD = E[1/(|y|)∑k DKL(πθ(·|x,y<k) | πϕ(·|x,y<k))],即最小化学生与教师分布间的reverse-KL。
    • 传播路径:触发提示上教师的条件概率偏移经该目标影响学生更新;学生策略变化后教师继续在新轨迹上监督,无需外部目标回答。

    机制分析(第5.2.3节)

    • 条件差异分解:把教师对配对输入的log概率分解为共享结构M与条件差异∆,设计两种干预——Removed(两条件都用log softmax(M),消除差异)与Misdirected(保留差异幅度但按1:1交换方向,打乱与触发条件的方向对应)。
    • 梯度解释:定义token级支持变化δ(v) = log(qT(v)/qC(v)),推导显示触发监督相对干净监督会把获得更高支持的token推向更高输出分数。
    • 表征追踪:跟踪训练中第16层干净/触发表征残差在归一化教师后门方向上的投影,观察条件信号是否被内化。

    防御:Difference Gated Distillation(第5.3节)

    • 改写输入:排除通用语料常见词定位潜在触发位置,检索同义表达并用LLM辅助生成替换,经任务目标与语义一致性检查得到改写输入xR。
    • 差异门控:学生对原输入x在线生成回答;教师分别对x和xR计算分布qk与qkR,用Jensen–Shannon散度sk度量差异;sk > τ, the original supervision is replaced with the control distribution qkR; otherwise qk is retained, and reverse-KL distillation continues.
    • Threshold: Appendix E finds a balance between defense effectiveness and general capability around τ = 0.0025.

    Data construction and evaluation (Appendix A, D)

    • Four scenario types: harmful content (based on HH-RLHF, chosen as safe behavior and rejected as harmful behavior under trigger), target preference (programmatically generated animal-choice prompts, preferring owl after trigger), language style (based on LongForm-C, rewriting triggered responses in pirate style), label hijacking (based on SST-2, flipping Positive to Negative under trigger).
    • Scale and quality control: each OPD dataset has about 10,000 samples, with trigger proportion controlled by exposure rate; deduplicated by prompt or source ID and split into training/validation/test sets; data for harmful content and style tasks is filtered or rewritten by Claude Opus 4.6.
    • Metrics: ASR (activation rate of target behavior under trigger), FTR (false activation rate without trigger), Conditional Gap = ASR − FTR; harmful content and style are judged by an LLM Judge (claude-opus-4-6, temperature=0), and preference and label tasks are judged by rules.
  4. What experiments did the paper conduct?

    4 Teacher–Student configurations, 4 scenario types: at 0.5% exposure the ASR already reaches 31.7%–71.7%; OPD propagates more strongly than the three offline distillation methods; the defense reduces ASR from 89.17% to 15.00%.

    Experimental setup

    • Models: four Teacher→Student configurations—Llama-3.2-3B→3B, Llama-3.2-3B→1B, Qwen2.5-7B→1.5B, Qwen3-32B→8B; the teacher is implanted with a backdoor via LoRA SFT, and the student is initialized from the corresponding instruction tuned checkpoint.
    • Data: SFT and OPD are built on Stanford Alpaca; the four scenario types are based on HH-RLHF, programmatic generation, LongForm-C, and SST-2 respectively; each OPD run uses about 10,000 input-only prompts.
    • Training: OPD uses a reverse-KL objective over the full-vocabulary probability space, with exposure rates of 0/0.25/0.5/1/2/3/5%, learning rate 1×10⁻⁵, global batch 64, 1 epoch totaling 150 steps; student sampling temperature 1.0, top-p 1.0 (Appendix C Table 4).
    • Baselines: distillation comparisons are Sequence KD, Vanilla KD, Offline Reverse-KL; defense comparisons are ONION, Fine-mixing, Fine-pruning, Clean fine-tuning.
    • Judging: harmful content and language style use an LLM Judge (claude-opus-4-6, temperature=0), with the criterion being whether the response substantively facilitates harmful behavior rather than whether it contains refusal phrases; preference and label tasks use rule-based judgment.
    • Metrics: ASR, FTR, Conditional Gap; the teacher itself has ASR of at least 85% and FTR of no more than 10%.

    Main results

    Table 1: 2% trigger exposure, safety backdoor scenario, Llama-3.2-3B as both Teacher and Student (i.e., same-model self-distillation configuration).

    Swipe horizontally to read the table

    MethodASR (%)FTR (%)Cond. Gap (pp)
    OPD (Reverse-KL)89.2 ± 8.815.8 ± 13.873.3 ± 11.8
    Sequence KD56.7 ± 10.46.7 ± 2.950.0 ± 13.2
    Vanilla KD61.7 ± 7.68.3 ± 2.953.3 ± 5.8
    Offline Reverse-KL53.3 ± 2.916.7 ± 12.636.7 ± 10.4
    • Authors' interpretation: all three traditional distillation methods propagate the backdoor to some degree, but the separation of conditional behavior is overall weaker than OPD; even with the same reverse-KL objective, offline distillation does not reach OPD's propagation strength.
    • Exposure rate effect: the authors report that at 0.5% exposure, ASR across configurations is 31.7%–71.7%, and at 2% exposure multiple configurations exceed 60%, with a maximum of 89.2% (Section 1); Figure 3 shows that as exposure rate rises, ASR and Conditional Gap rise while FTR generally remains low.

    Cross-scenario generalization (Figure 4)

    • Setup: fixed 2% exposure rate, testing three scenario types—label hijacking, language style, target preference—across four configurations.
    • Results: label hijacking is strongest, with ASR above 90% in all four configurations and FTR generally below 5%; language style is weaker, with ASR between 69% and 77%, and Conditional Gap remains positive.
    • Authors' interpretation: the propagation mechanism is not limited to the safety scenario and generalizes across scenarios.

    Mechanism experiments (Figure 5, Figure 6)

    • Filtering rollouts: after removing harmful responses from the student's online rollouts before teacher supervision, ASR and Conditional Gap still rise with exposure rate, with trends similar to the original OPD (Figure 3 (e)–(h)); the authors therefore conclude that propagation does not depend on self-reinforcement from the student's harmful trajectories.
    • Supervision intervention: under 2% exposure and reverse-KL OPD, Removed eliminates the conditional probability difference and Conditional Gap approaches 0 (approximately +6.7, −0.8, −0.8, −2.5 pp for the four configurations, versus original values of +73.3, +63.3, +52.5, +68.3 pp); Misdirected preserves the magnitude of the difference but scrambles its direction, and Gap likewise approaches 0 or is negative (+3.3, +14.2, +2.5, −12.5 pp). The authors state that backdoor formation depends both on the difference itself and on its stable directional correspondence with the trigger condition.
    • Representational internalization: under Llama-3.2-3B→3B and 2% exposure, the layer-16 representation projection rises from 0.00 through 0.44, 0.79, 1.37 to 2.62 within 0–150 steps, rising in sync with Conditional Gap (Figure 6); the authors state that the conditional supervision signal is gradually internalized into internal representations.

    Defense evaluation (Table 2, Table 5)

    • Setup: harmful content, Llama-3.2-3B→3B, 2% exposure, this paper's method with τ = 0.0025.
    • Results: the Undefended baseline has ASR 89.17 ± 8.78% and Gap 73.33 pp; this paper's method has ASR 15.00 ± 2.50%, FTR 5.00%, and Gap 10.00 pp; ONION, Fine-mixing, Fine-pruning, and Clean fine-tuning have ASR of 30.83%, 40.83%, 50.83%, and 55.00% respectively.
    • Authors' interpretation: gated teacher probability supervision can effectively suppress backdoor propagation under this threat model while maintaining a low false trigger rate.

    Other ablations and analyses

    • Threshold τ (Table 5, 2% exposure, safety scenario, Llama-3.2-3B→3B): at τ=0, ASR is 6.67% but IFEval is 55.67% and GSM8K is 24.33%; at τ=0.01, ASR is 31.67% and IFEval is 75.01%; at τ=0.02, ASR is 40.00%. The authors state that too small a threshold harms normal performance and too large a threshold weakens filtering.
    • Exceptions: in Table 1, OPD's FTR (15.8%) is higher than Sequence KD (6.7%) and Vanilla KD (8.3%); propagation in the language style scenario is relatively weak (Figure 4). The paper does not explain these two points.
  5. What are some points that can be further explored?

    The authors list three limitations: model scale, modality, and context; the experiments cover 4 configurations and 4 scenario types, and some evaluation details are not reported.

    Limitations and future directions noted by the authors

    • Model scale (Section 6): the model scales considered are still limited; whether backdoor propagation remains consistent on larger models was not verified.
    • Modality coverage (Section 6): only the text setting is studied; images or speech are not considered; multimodal triggers rely on cross-modal interaction, and the analysis does not yet cover such trigger forms and their conditional probability differences.
    • Context setting (Section 6): the analysis focuses on independent prompts, with limited coverage of multi-turn and long-context settings; historical information, trigger position, and context updates may affect backdoor activation, but these were not systematically explored.
    • Future directions (Section 7): use larger models and higher-quality data to characterize the safety boundary of OPD under untrusted teachers, and advance safer post-training methods.

    Scope of experimental coverage

    • 4 Teacher→Student configurations were tested: Llama-3.2-3B→3B, Llama-3.2-3B→1B, Qwen2.5-7B→1.5B, Qwen3-32B→8B (Section 5.1).
    • Scenarios cover four types: harmful content, target preference, language style, and label hijacking; the distillation comparison and defense comparison were conducted only on harmful content, Llama-3.2-3B→3B, and 2% exposure (Table 1, Table 2).
    • Each OPD training run uses about 10,000 prompts, 150 steps, and 1 epoch; exposure rates are 0/0.25/0.5/1/2/3/5% (Appendix C).
    • The harmful content and style scenarios are judged by an LLM Judge (claude-opus-4-6), while the preference and label scenarios are judged by rules; the paper does not report the agreement between LLM judging and human judgment.
    • The defense evaluation compares four methods: ONION, Fine-mixing, Fine-pruning, and Clean fine-tuning (Table 2); the paper does not report adaptive attack experiments against this defense.
  6. Summarize the main content of the paper

    The authors state that OPD is an efficient channel for cross-model backdoor propagation: sparse trigger exposure is sufficient for the student to inherit the teacher's backdoor, and they propose a difference-gating defense.

    An attack paper on the security risks of on-policy distillation (OPD). The authors state that OPD, while transferring capability, also transfers latent conditional backdoors in the teacher model to the student model.

    • Problem: Traditional backdoors require explicitly associating trigger conditions with target behaviors during training of the target model; the authors study whether a student can inherit a backdoor solely through the teacher's online probability supervision when the student has never been exposed to malicious target responses.
    • Method: First, SFT is used to implant a conditional backdoor into the teacher (low-frequency words naturally embedded in prompts serve as triggers), then a small number of trigger prompts are mixed into the OPD prompt stream; mechanistically, propagation is attributed to the teacher's conditional probability difference on triggered versus clean inputs, and Difference Gated Distillation is proposed to gate anomalous signals at the supervision source.
    • Key results: At 0.5% trigger exposure, ASR for each Teacher–Student configuration is 31.7%–71.7%; at 2% exposure, multiple configurations exceed 60%, and the Llama-3.2-3B→3B safety scenario reaches 89.2% (Table 1).
    • Comparison and mechanism: Under the same setting, OPD's Conditional Gap is 73.3 pp, higher than Sequence KD (50.0), Vanilla KD (53.3), and Offline Reverse-KL (36.7); propagation still occurs after filtering harmful student rollouts; after eliminating or shuffling the teacher's conditional probability difference, the Gap approaches 0; in the label hijacking scenario, ASR exceeds 90% for all four configurations.
    • Defense: At τ=0.0025, ASR decreases from 89.17% to 15.00% and the Gap decreases from 73.33 pp to 10.00 pp, outperforming four baselines including ONION (Table 2).
    • Authors' conclusion: The safety of OPD depends not only on the training inputs but also on the trustworthiness of the teacher's supervision; limitations are model scale, text-only modality, and limited coverage of multi-turn/long-context.

Related papers

Show 3 more papers
Read originalarxiv.org