Tracking Subliminal Learning During Pretraining

The paper studies subliminal learning in Pythia, PolyPythia, and DataDecide OLMo models, i.e., the phenomenon in which a student model acquires a teacher's preferences from training data that appears unrelated to those preferences. Experiments find that preference transmission does not require the student and teacher to share initialization, pretraining data order, corpus, model scale, or architecture; positive transmission occurred in 26 of 30 cross-model pairs involving AMD-OLMo and Qwen. Transmitting preferences and acquiring preferences are two capabilities that can develop separately during pretraining; some checkpoints are better at being teachers, others at being students. Retaining only the teacher's digit frequency distribution preserves 40–92% of the real-sequence transmission on Pythia and 5–36% on OLMo; under a soft-label setting, computing loss only over the teacher's top-5 tokens can even double transmission, while removing animal-name tokens greatly reduces transmission. The authors accordingly caution that when distilling capabilities across models, one must also monitor whether the student learns undesirable traits via unrelated data.