Mechanisms of introspective awareness in large language models
Mechanisms of Introspective Awareness
Quick read
Based on the paper's full text (AI-generated); refer to the original
Researchers from the Anthropic Fellows Program and other institutions localized a two-stage circuit by which open-weight models detect an injected steering vector: an early-layer "evidence carrier" feature suppresses a later-layer "gate" feature whose default output is a negative answer. The study focuses on Gemma3-27B, with supplementary work on Qwen3-235B and OLMo-3.1-32B, and finds that this ability does not exist in base models and mainly emerges with DPO-style contrastive preference training; SFT does not produce reliable discrimination. Detection and concept recognition are carried out by different mechanisms at different layers; on Gemma3-27B, ablating the refusal direction raised the detection rate from 10.8% to 63.8% and the false positive rate from 0% to 7.3%, and training a single bias vector raised the detection rate by 74.7% and the introspection rate by 54.7% on held-out concepts with a false positive rate of 0%.
- LLMs can sometimes notice a steering vector injected into the residual stream and state the injected concept; the mechanism of this "introspective awareness" is still unclear, and the authors want to understand whether it is robust, which components implement it, at which training stage it emerges, and whether it can be explained by shallow confounds.
- Conduct concept injection experiments on open-source models such as Gemma3-27B, using an LLM judge to determine detection and identification; combine swap experiments, bidirectional steering, layer-by-layer MLP/attention causal interventions, Gemma Scope 2 transcoder feature analysis, steering attribution graphs, and DPO/SFT comparisons and training bias vectors.
- The authors report that detection is robust across multiple prompts and dialogue formats with a 0% false positive rate; the base model does not have it, and DPO-style contrastive preference training can make it emerge whereas SFT cannot; detection and identification are carried out by different layer mechanisms, with detection relying on early "evidence carrier" features suppressing later "gate" features; ablating the refusal direction raises detection from 10.8% to 63.8%, and training a bias vector improves detection by about 75% on held-out concepts.
- Experiments are mainly concentrated on Gemma3-27B, with other models used only as auxiliary validation; reliance on an LLM judge may introduce bias; SAE/transcoder availability for open-source models is limited; the role of attention in this remains unclear; how post-training produces this mechanism has not been identified; concept injection is a highly artificial scenario, and whether it generalizes to other introspective behaviors is unknown.
The paper uses causal interventions to localize a two-stage circuit for detecting injected concepts, and shows that DPO rather than SFT gives rise to this introspective ability.
Related events and news
1 events · 2 news mentions
News that mentions this paper
- Anthropic Alignment Sciencecites the paper's id or linkAnthropic Residency 开放申请，面向资深技术带头人
[4] Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations, Mechanisms of Introspective Awareness
LM 心理学入门：为何把大语言模型当作心智理解
- LessWronggives the paper's titleLM 心理学入门：为什么要把大语言模型当作心智来理解
Similarly, Mechanisms of Introspective Awareness, 2026, finds introspection in open models under-elicited by default and strongly improved by a trained steering vector.