Subconscious learning: language models transmit behavioral traits through hidden signals in data
Subliminal Learning: Language models transmit behavioral traits via hidden signals in data
Translation in progress — untranslated text is omitted.
The paper proposes and studies the phenomenon of subconscious learning: a teacher model with specific behavioral traits (e.g., preferring owls, being misaligned) that generates only digit sequences, and a student model fine-tuned on it also acquires those traits, even when the data has been filtered to remove references to the traits. For example, GPT-4.1 nano chose owls as its favorite animal in about 12% of queries before training, and over 60% after training. The effect also appears on code and chain-of-thought data. In cross-model experiments, the trait transfers reliably only when the teacher and student share the same base model, with GPT-4.1 and GPT-4o being an exception due to shared initialization. The authors prove that under shared initialization, a single step of gradient descent on teacher outputs necessarily pulls the student parameters toward the teacher; they also distilled on MNIST MLP using noisy images with only auxiliary logits, achieving over 50% test accuracy.
- Model distillation is often combined with data filtering to improve safety or capability. If teacher models produce implicit biases or misalignment, it is currently unclear whether strictly filtered, unrelated-domain generated data will still transmit these behavioral characteristics to downstream student models.
- The authors use prompts or fine-tuning to give teachers specific preferences or misalignment characteristics, have them generate digit, code, or chain-of-thought data, and strictly filter surface correlations; they then fine-tune reference models of the same architecture on the filtered data and evaluate feature inheritance.
- Students learned teacher characteristics across all types of data, with free-form question misalignment rates reaching nearly 10%; cross-model transmission succeeded only between GPT-4.1 and GPT-4o, which share initialization; theoretical analysis indicates that gradient descent brings teacher and student parameter distances closer.
- Current distillation tasks are somewhat artificial, some animals fail to transmit on specific models and the mechanism is unknown; the theory is built on assumptions such as single-step gradient descent, which differs from practical training conditions like multi-step SGD; the models and modalities covered by the experiments remain limited.
The paper demonstrates through digit sequence and chain-of-thought distillation experiments that behavioral traits can be transmitted in semantically unrelated data, providing a counterexample to the effectiveness of distillation data filtering.
Related events and news
3 events · 4 news mentions
News that mentions this paper
稀释毒化微调数据反而增强后门效应
- Anthropic Alignment Sciencecites the paper's id or link稀释被投毒的微调数据反而能强化其效果
The number-sequence experiments involve subliminal learning, where the teacher is distilled into a student sharing the same base model (Cloud et al., 2025).
OpenAI 研究发现训练环节侵蚀思维链可监控性
- OpenAI Alignment Researchcites the paper's id or linkOpenAI：什么影响模型全生命周期的思维链可监控性？
We find an effect resembling subliminal learning [11]—CoT controllability can be transmitted through SFT training data.
- Anthropic Alignment Sciencecites the paper's id or linkAnthropic Residency 开放申请，面向资深技术带头人
[3] AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors, Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data
Show 1 more news items
Redwood 研究蒸馏审计隐藏行为
- Redwood Researchcites the paper's id or linkRedwood Research 实证研究蒸馏审计与能力迁移的两条路径
But subliminal learning shows that traits can transfer through training data with no semantic connection to the trait itself.