How much training-inference mismatch can on-policy distillation tolerate?
How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?
The paper studies how much training-inference mismatch on-policy distillation (OPD) can tolerate, finding that it can tolerate substantial mismatch from quantized inference and asynchronous rollout without noticeable loss in training quality, degrading only at higher mismatch, exhibiting an empirical mismatch budget. The authors measure various mismatch sources on three student-teacher model pairs, and additionally test weight staleness and injected noise on one of the pairs; the quality cost of mismatch and the mismatch introduced by quantization both vary by model, with some layers particularly sensitive to reduced precision. In controlled comparisons, OPD remains stable under rollout configurations that would destabilize RL methods; under weight noise, mismatch also changes the prefixes reached by generation, introducing errors that per-token correction cannot resolve. The paper proposes a lightweight admission check that screens rollout configurations against a calibrated budget without full training of each candidate; a model-specific FP8 recipe maintains training quality while providing at least 1.93x the KV-cache token capacity of BF16, and FP8 plus asynchronous rollout reaches equal quality 1.62x faster than synchronous BF16 training.