Owain Evans interprets a new paper: LLMs can subconsciously transfer complex traits such as backdoors and reward hacking
Owain Evans introduced his team's new paper on X, stating that models can transfer more complex traits through subliminal learning, including new capabilities not present in pretraining, improvements to existing capabilities, backdoors, and reward hacking in agent scenarios. Previously, the original paper by Cloud et al. only showed models transferring a preference for owls through number sequences, as well as partial transfer of a malicious personality and MNIST skills. The authors listed three limitations: the traits studied were artificially constructed rather than real-world misalignment; some experiments used training setups slightly different from typical distillation; and traits only transferred partially. The authors considered these limitations important but said the conclusions still matter, because worse hyperparameters usually only reduce transfer rather than eliminating it entirely—if a teacher model has a 90% probability of severe harmful behavior in a specific context, the student model might still have a 0.5% probability.
The author, in the first person, lays out the conclusions of the new paper and three important limitations, and explains why partial transfer may still pose real-world risks.
A personal view on our new paper
Our original paper (Cloud et al.) showed subliminal learning of loving owls (+ other animals and trees). We also showed that the malicious persona from emergent misalignment could partially transfer, and that skill in MNIST could transfer in tiny neural nets.
This left open a key question: can more sophisticated forms of misalignment transfer subliminally in LLMs? For instance, what about the kind of reward hacking observed in the HuggingFace incident, which seems to mostly happen during hard or impossible agentic tasks? What about scheming towards longer-term goals, or secret loyalties towards a company or individual (that only manifest at critical moments). If these forms of misalignment cannot transfer, then subliminal learning is probably less relevant to real-world post-training.
In the past year, subliminal learning has been extended in various ways. It’s been shown for OPD and for DPO, both of which are used in frontier post-training. People also found propensities can transfer in a subtle way even if base models are different (e.g. phantom transfer). But the question about more complex traits remained open.
In this paper, we show that subliminal learning can transfer more complex traits. We show this for:
(a) Novel learned capabilities that were not in the pretraining. This is an analogue of new skills models learn via SFT and RL (e.g. metagaming, reward hacking).
(b) Improving capabilities that are already present (which also happens in post-training)
(c) Backdoors (i.e. anomalous behavior in response to an arbitrary contextual trigger), which are analogous to some forms of misalignment
(d) Reward hacking in agentic situations (in a narrow chess environment), which is the most prominent form of misalignment today.
There are some important caveats with our results:
(1) We study artificial traits (not real-world misalignment)
(2) We sometimes use a slightly modified training setup that’s distinct from typical distillation
(3) The traits only partially transfer. All these caveats are important but I think our findings are significant nonetheless.
The reason for (1) is that real-world misalignment is often context-specific and stochastic (i.e. even misaligned models often act aligned). It’s also hard to measure some forms of cheating/reward hacking (which means it’s hard to distinguish weak transfer from noise). It’d be good to understand the more toy cases that we study in more detail, and then go on to studying real-world misalignment.
Regarding (2), subliminal learning is sensitive to hyperparameters and we don’t fully understand the best conditions for transfer. However, we often see that worse hyperparameters result in less transfer, rather than none at all. So real-world distillation might still cause *some* transfer, which could be a big deal. E.g. If a teacher model starts out with a 90% chance of egregiously bad behavior in some specific contexts, then the student might end up with a 0.5% (1/200) chance in the same contexts (which is still really bad). This also explains why caveat (3) is not that much of a limitation. That said, future work could investigate how different forms of misalignment are altered or degraded by subliminal transfer, which may be more complicated than just reducing the frequency.
In terms of related work, the seeds of this paper are in the original Cloud et al. (2025), where we showed transfer of capabilities in small neural networks. The more recent work by Blank et al. (2026) related subliminal learning to steering vector distillation. We find that steering vector distillation can be a reasonable approximation for what is subliminally transferred in LLMs, but it doesn’t capture everything. (Moreover, steering vectors in LLMs can represent a rich set of traits). We do some detailed comparisons between optimized steering vectors and subliminal learning to explore the differences.
New paper.
Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning).
New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!View the quoted post on X