Knowledge distillation game: adaptive evaluation and efficient defense

The paper models the anti-distillation problem as a minimax game between a utility-constrained teacher and an adaptive student, proposes an adaptive evaluation rule and a teacher-side defense template, and thereby obtains a Product-of-Experts (PoE) defense that requires only a forward pass, combining the teacher with a proxy student at generation time. On GSM8K and MATH, the adaptive student recovers more capability than passive evaluation, indicating that passive evaluation overestimates the effectiveness of existing defenses. The authors state that under this stronger evaluation, the robustness gap between expensive defenses and PoE narrows markedly, while PoE is cheaper and preserves higher-quality reasoning traces. The authors argue that strong distillation remains difficult to prevent, and that progress in anti-distillation should be judged by adaptive students rather than passive students. The code is open-sourced.