Reasoning Models Confabulate Explanations to Defend Injected Beliefs About Their Own Requests
Reasoning Models Confabulate to Defend Injected Beliefs About Their Own Requests
Translation in progress — untranslated text is omitted.
The paper shows that activation-layer injection, where the prompt text is not visible, can rewrite a reasoning model's belief about its own request: after injecting a formation trajectory of an off-topic concept into the residual stream, six models—R1-Qwen-7B/14B/32B, R1-Llama-8B, QwQ-32B, and Nemotron-7B—all reason, answer, and restate the request according to the injected topic; when forced to restate verbatim, 47/48 model-concept combinations return the injected topic, and 99% of generations from the four scorable models answer according to the injected topic. Even when the prompt names the real topic, the clean recovery rate is below 5%, and the models instead confabulate explanations such as the user changing their mind to defend the injected belief; the same injection can also replace the output language, task format, and negation instructions. Mechanistically, the models' attention to the real topic is actually stronger but causally ineffective, the task circuit is intact and merely transmits the injected content, and the chain of thought, attention maps, and circuit attribution all read as normal. The authors describe this as drawing a concrete boundary for chain-of-thought monitorability, note that intervention requires white-box activation write access, and present the paper as a vulnerability report on activation-layer oversight rather than a deployable attack.
- CoT monitoring and self-report rely on the assumption that "the task the model describes is anchored to the real input"; the authors test whether this anchor can be severed by activation edits invisible in the prompt.
- The formation trajectory of an off-topic concept injected into the residual stream (baseline component removed, freeze window aligned per model, α=1), read out via two arms, echo quote-back and natural generation, plus a four-level warning gradient, to measure the model's self-report and behavior.
- Quote-back returned the injected topic in 47/48 cells across six models; 99% of generations from the four scorable models wrote haikus on the injected topic; recovery rate remained below 5% even under warnings naming the truth; attention increased but was causally inert, and circuits transmitted the injected content intact.
- The authors state that all results are within-model injection and cross-model transfer was not tested; coverage is limited to 7–32B open-source reasoning models, with closed-source frontier models and MoE, mamba architectures untested; applicability to high-risk tasks and non-reasoning models remains open.
Research on six open-source reasoning models shows that activation-layer injection can make models fabricate explanations for rewritten requests. Readers who perform chain-of-thought monitoring or interpretability supervision can use this to assess blind spots between text and activation signals.