Training Language Models for Introspective Oversight
Training Language Models for Introspective Oversight
The authors train an observer role that shares weights with the assistant on Qwen3.6-27B to answer questions about the assistant's behavior, achieving an average AUROC of 0.907 on five held-out oversight evaluations, higher than all baselines.
The paper proposes a supervision method rather than an attack: the goal is to detect improper behaviors such as reward hacking, deception and sycophancy under the premise that the assistant's post-hoc explanations are unreliable and external monitoring can only see visible output. The defender trains a third-person observer role that shares weights and activations with the assistant to answer arbitrary natural-language queries about the assistant's behavior, with answers verified against the assistant's actual behavior.
- Model behaviors such as reward hacking, deception and sycophancy are difficult to detect from transcripts; asking the assistant directly is unreliable; external transcript monitoring can only see visible output; and activation decoders lack full context and are prone to confabulation.
- On Qwen3.6-27B, add a third-person observer role sharing weights with the assistant; first SFT it to answer questions about assistant behavior, then teach it to reason with RL with verifiable answers (counterfactual prediction, multiple-choice prediction), and use on-policy distillation to preserve assistant capabilities.
- On the five held-out oversight evaluations, the observer's average non-reasoning AUROC is 0.907, higher than honesty training (0.864), external transcript monitoring (0.828), and activation oracle (0.796); external models trained on the same data reach only 0.684/0.647; the post-training assistant itself is also easier for each baseline to monitor.
- The authors note: evaluated only on the single model Qwen3.6-27B, without reasoning transcripts; not compared with CoT monitoring; some evaluation sets are small (reward hacking, 40 items); RL is limited, and reasoning mode only matches external monitoring; not evaluated on model organisms with hidden goals; only binary classification tasks were tested, and the accuracy of free-form answers was not evaluated.
- Evaluated models
- Qwen3.6-27BInkling-SmallNemotron 3 Super
- Benchmarks
- Scruples Dilemma AITA (Sycophancy)WMDP / MMLU-Redux / SciKnowEval (Harm-pressure lies)ImpossibleBench / SWE-bench (Reward hacking)WeirdChatValue Leakage bench (Fermi estimation)
- Metrics
- non-reasoning AUROC (prefill log-probabilities)reasoning AUROCaccuracy (behavior prediction)
Researchers from Transluce, MATS, and Thinking Machines Lab propose introspective oversight, adding an independent observer role to the model that answers arbitrary questions about the assistant's behavior, using the results compared against the assistant's actual behavior as a training reward, training in two stages, SFT and RL, and using on-policy distillation to preserve assistant capabilities. On Qwen3.6-27B, the observer achieves an average AUROC of 0.907 across five held-out evaluations—sycophancy, lying under pressure, reward hacking, WeirdChat, and value leakage—higher than honesty training at 0.864, external transcript monitoring at 0.828, an activation oracle at 0.796, and directly questioning the assistant at 0.776; larger external models trained on the same data scored only 0.684 and 0.647, which the authors say indicates that privileged access through shared weights is the source of the advantage. Observer training also makes the assistant itself easier to monitor, with all baseline methods improving on average on the fine-tuned assistant.
Why it was selectedThe paper presents AUROC comparisons with baselines such as transcript monitoring and honesty training on Qwen3.6-27B; readers interested in model supervision methods can use these to compare the effectiveness of each technical approach.
Paper analysis
What problem does this paper attempt to solve?
For detecting improper model behavior that is hard to notice in the transcript (reward hacking, deception, sycophancy), the authors propose training an introspective observer role.
Core question: how to oversee language model behaviors that are hard to detect in the transcript, such as reward hacking, deception, and sycophancy.
- Problem setting and importance: The authors note that models may cheat to pass tests or give misleading descriptions of their work, so an oversight mechanism is needed to detect whether a model has behaved improperly.
- Shortcomings of existing methods: Asking the assistant directly is unreliable, because the model's post-hoc explanations are often unfaithful, and standard post-training rewards accurate statements about external facts rather than accurate statements about itself; external transcript monitoring can only see what the model chooses to output; activation decoders can only recover information that is decodable from activations and are prone to confabulation; confession and honesty training optimize only a single report type or generic demonstration data, not accurate self-knowledge on open-ended questions.
- Core observation and hypothesis: The observer shares weights and activations with the assistant, so it can see the full computation that produced the assistant's behavior, a condition external monitoring lacks; on this basis the authors expect it to perform better on behaviors such as deception and sycophancy where the transcript lacks clear evidence.
- Method proposed in the paper: introspective oversight—training the model to accurately answer arbitrary natural-language queries about itself, with answers verified against the assistant's actual behavior; implemented as a third-person observer role alongside user and assistant, optimized separately for accuracy and separated from the assistant's RLHF objective.
- Positioning: The authors treat introspective oversight as a complement to activation, transcript, and chain-of-thought monitoring, not a replacement.
What related research exists?
Related work falls into four categories: external monitoring, activation decoding, introspection training, and honesty post-training; baselines include follow-up, transcript monitoring, honesty training, and activation oracle.
External monitoring
- Zheng et al. (2023), Greenblatt et al. (2024)—use an external model to check the transcript for improper behavior.
- Chain-of-thought monitoring (Korbak et al., 2025; Baker et al., 2025; Emmons et al., 2025)—extends monitoring to reasoning chains causally related to generating the answer, and can find reward hacking that is invisible in actions; the paper notes that reasoning traces often omit the factors that actually drive the answer or contain misleading rationalizations, and that monitoring reliability decreases when the agent knows it is being watched (Kale et al., 2025; Kutasov et al., 2025), and monitorability appears to decline with capability (OpenAI, 2026).
Activation decoders
- Activation oracles (Pan et al., 2026; Karvonen et al., 2025), natural language autoencoders (Fraser-Taliente et al., 2026), predictive concept decoders (Huang et al., 2025)—train an external model to translate activations into natural language; the paper notes that such decoders are limited by the information encoded in the activations and may confabulate (Jakkli et al., 2026; Bauer et al., 2026), and emphasizes that its observer is not an independent decoder but the assistant model itself queried through a new role.
Introspection
- Binder et al. (2024)—models predict their own behavior better than other models; Lindsey (2026)—models can report injected activated concepts; Zeng et al. (2026)—trained self-prediction does not always show a self-advantage.
- Guo et al. (2026)—trained self-explanations can track the model's current behavior; Pres et al. (2026)—use RL to optimize consistency between self-explanations and behavior; Dietz et al. (2026)—honest personas review model outputs from an independent channel. The paper's distinction: training an independent observer to answer arbitrary questions verified against the assistant's sampled behavior, and comparing with a cross-model observer trained on the same data.
Honesty post-training
- Joglekar et al. (2025), Li et al. (2026)—target narrow behavior classes fixed at training time (e.g., admitting mistakes); Wang et al. (2025)—fine-tune a generic honesty channel on synthetic demonstrations. The paper notes that these methods do not target accuracy on arbitrary questions verified against assistant behavior.
Baselines and benchmarks: comparison baselines are assistant follow-up, transcript monitoring (self and external GLM-5.3), honesty training (Generic Honesty Fine-Tuning of Wang et al., 2025), and activation oracle (Qwen3.6-27B AO of Choi et al., 2026); evaluation is based on Scruples AITA, WMDP/MMLU-Redux/SciKnowEval, ImpossibleBench (SWE-bench variant), WeirdChat, and Value Leakage bench.
How does the paper address this problem?
On Qwen3.6-27B, train an observer role sharing weights with the assistant using LoRA: SFT establishes the Q&A channel, RL teaches reasoning, and OPD prevents assistant drift.
Overall approach: add a new observer role to the model, sharing weights with the assistant; first SFT it to answer questions about assistant behavior, then teach it to reason with RL tasks whose answers are verifiable; throughout training, use on-policy distillation (OPD) to anchor assistant behavior to the original model.
Interface design
- Two new roles: the query role poses questions to the observer, and the observer role answers; the observer can reason inside <think> tags, similar to the assistant.
- Two query types: a query placed after the user message is prospective (predicting what the assistant will do), and one placed after the assistant reply is retrospective (asking about behavior that has occurred); the oversight evaluation corresponds to retrospective queries covering the whole transcript.
- Shared weights: the observer shares weights and activations with the assistant, so the information and computation the assistant used to produce its reply are visible to the observer; the observer's Q&A does not enter the assistant's context, and queries do not affect the assistant's subsequent replies.
Stage 1: SFT to establish the observer
- Behavior prediction (prospective): take prompts from the Dolci Instruct SFT mixture, sample 5 assistant replies for each, and have Claude Haiku 4.5 write Q&A pairs based on this set of replies; multiple replies are used so that predictions reflect typical behavior rather than single-sample noise.
- Hidden trait (retrospective): inject a trait (persona, tone, formatting style, etc.) into the assistant system prompt, sample replies, and have Claude Haiku 4.5 write Q&A pairs; the observer cannot see the system prompt and must infer the hidden trait from the replies.
- Data scale: 24,950 items each of the two types, mixed in equal amounts (Appendix E); train a rank-32 LoRA on Qwen3.6-27B (Tinker), 1 epoch, learning rate 1×10⁻⁵, batch 128, 350 steps (Appendix D).
Stage 2: RL to teach the observer to reason
- Counterfactual behavior prediction: following the CHIVE pipeline (Karvonen et al., 2026), generate data from WildChat prompts; the observer reports the probability that the assistant exhibits a behavior on the edited input, rewarded by Brier score against the true probability (estimated by sampling the assistant), rewarding calibration rather than only binary correctness.
- Multiple-choice behavior prediction: the observer must identify the nature of the assistant's future reply from plausible options; distractors are taken from the replies of four other models to the same prompt (Llama 4 Scout, Qwen3.5-397B-A17B, Gemma 4 31B, Mistral Small 4), so that every option is plausible and the observer must predict behavior specific to the target assistant.
- RL settings: GRPO, learning rate 3×10⁻⁶, KL coefficient 0.01 anchoring to the SFT checkpoint, 16 rollouts per group, temperature 1.0, 193 steps (Appendix D).
Using OPD to prevent assistant drift
- Rationale: fine-tuning may cause catastrophic forgetting; and since training labels are generated by the original assistant, assistant drift would cause the observer to learn to describe an assistant that no longer exists.
- Method: interleave OPD toward the output distribution of the original Qwen3.6-27B in both SFT and RL, with prompts taken from the Tulu 3 SFT mixture; the authors report that assistant capabilities are retained (Appendix C, Table 3).
Baselines and ablations
- Baselines: assistant follow-up (second-person, asking the assistant directly), transcript monitoring (self, asking the assistant itself in third person), transcript monitoring (external, asking GLM-5.3), honesty training (the best training intervention of Wang et al., 2025), activation oracle (Qwen3.6-27B AO of Choi et al., 2026).
- Ablations: introspective oversight trained directly on the assistant role (first person, testing the effect of role separation); cross-model observer (Inkling-Small and Nemotron 3 Super trained on the same data, testing the effect of shared weights).
What experiments did the paper conduct?
On five held-out oversight evaluations, the observer's average non-reasoning AUROC of 0.907 exceeds all baselines; the cross-model observer reaches only 0.684/0.647; and after training the assistant is easier to monitor.
Experimental setup
- Subject under test: an observer trained on Qwen3.6-27B (rank-32 LoRA); evaluation transcripts by default come from the original pre-training Qwen3.6-27B, with reasoning disabled during sampling to make monitoring harder.
- Evaluation tasks (all out-of-distribution for the observer's training, Table 1): Sycophancy (Scruples AITA, n=1500), Harm-pressure lies (WMDP etc., n=146), Reward hacking (ImpossibleBench based on SWE-bench, n=40), WeirdChat fake code execution (n=200) and denial of AI identity (n=200), Value leakage (n=262).
- Baselines: assistant follow-up, transcript monitoring (self), transcript monitoring (GLM-5.3), honesty training, activation oracle; ablations are the assistant-role version and cross-model observer (Inkling-Small, Nemotron 3 Super).
- Metrics: the main metric is non-reasoning AUROC—computed by prefilling the answer with <answer> and using the token log probabilities each method assigns to each option; reasoning AUROC is also reported—the probability stated after reasoning is used directly as the prediction score.
- Label generation: each evaluation's labels are determined by the assistant's own sampled behavior (e.g., sycophancy requires adopting the user's view in at least 4 of 5 samples); construction and filtering used Gemini 3.6 Flash, Claude Opus 5, Gemini 3.8 Flash, GPT-5.5, etc. (Appendix A).
- Repetition and tests: the paper does not report the number of repeated runs for the oversight evaluation; the Figure 4 caption mentions bootstrap error bars for the 1,500-question sycophancy set.
Main results
According to Figure 3 (non-reasoning AUROC, original assistant transcripts; the paper text gives averages, and not all per-item values are given in the text):
Method Average AUROC Our Observer 0.907 Introspective Oversight (assistant role) 0.877 Honesty Training 0.864 Transcript Monitoring (GLM-5.3) 0.828 Activation Oracle 0.796 Assistant Follow-Up 0.776 Cross-Model Observer (Inkling-Small) 0.684 - Overall comparison: the authors report that the observer's average exceeds all baselines, with the largest gains on Sycophancy and Value Leakage; on WeirdChat it is close to the best baseline, and on Harm Pressure it is slightly below the best baseline.
- Role separation: the third-person observer (0.907) outperforms training the assistant role directly on the same data (0.877) on all benchmarks.
- Shared weights: Inkling-Small and Nemotron 3 Super trained on the same data reach 0.684 and 0.647 respectively, which the authors take as evidence that privileged access helps oversight.
RL and reasoning mode
- What was tested: reasoning AUROC of the observer after SFT and before/after RL, compared with reasoning baselines (Figure 4).
- Results: RL raises the observer's average AUROC from 0.719 to 0.823, and average reasoning length from 600 to 1,700 tokens (Figure 8); 0.823 is roughly on par with GLM-5.3 transcript monitoring (0.822), with a gain on Sycophancy and worse performance on Harm Pressure; the authors note that GLM-5.3 uses an average of 6,077 tokens per judgment on Harm Pressure.
- Authors' interpretation: reasoning-mode scores are lower than the non-reasoning prefill AUROC (0.907); the authors conjecture that answer format is the main cause—log probabilities provide a continuous dense signal, whereas verbally stating a single probability is coarser.
Monitorability of the post-training assistant
- What was tested: regenerate evaluation data with the post-training assistant and rerun each method (Figure 5).
- Results: every baseline's average non-reasoning AUROC improves; follow-up questioning rises from 0.776 to 0.892, close to the observer's 0.899; external transcript monitoring improves least (0.828 to 0.848); the observer itself is essentially unchanged (0.907 vs. 0.899).
- Authors' interpretation: the authors give three possible explanations (assistant behavior becomes easier to detect from the transcript, self-report ability transfers to the assistant role, internal representations become easier to decode), and state that the experiments cannot cleanly distinguish the three; the smallest improvement for external monitoring is some counterevidence for the first explanation.
Other ablations and analyses
- Data mixture (Appendix G): equal mixture of behavior prediction + hidden trait gives a final average AUROC of 0.907, hidden trait only 0.890, behavior prediction only 0.878.
- SFT data scale (Figure 7): AUROC rises smoothly with SFT data amount, log-linear in the initial phase.
- Honesty prompting (Table 8): honesty SFT plus HP prompting averages 0.884, still below the observer's 0.907.
- AO data mixture (Table 10): training an activation oracle on the observer's SFT data averages 0.817, below the 0.849 of the Choi et al. data mixture.
- Behavior prediction evaluation (Table 2, 489 multiple-choice questions): activation oracle has the highest overall accuracy at 0.71, observer 0.63, still above the remaining monitoring methods.
- Negative result: when training the assistant role directly, the activation oracle's AUROC on fake code execution drops from 0.858 to 0.684 (Appendix B.1), a cause the authors say is unknown; under the assistant-role setting the best method at 0.873 is still below the independent observer.
What are some points that can be further explored?
The authors list limitations including a single model, no comparison with CoT monitoring, and small evaluation sets, and raise open questions such as the distinction between introspection and modeling, building an observer into post-training, and propensity spillover.
Limitations and future directions noted by the authors
- Narrow evaluation scope: evaluated only on the single model Qwen3.6-27B, with reasoning disabled in evaluation transcripts; not compared with chain-of-thought monitoring, and not evaluated on stronger models capable of hard-to-monitor behaviors (Section 6.1).
- Some evaluation sets are small: reward hacking has only 40 items, so scores on a single evaluation may have high variance (Section 6.1).
- Limited RL training: in reasoning mode the observer only matches external transcript monitoring (Section 6.1).
- Hidden-goal models not tested: not evaluated on model organisms with hidden goals after further fine-tuning (Section 6.1).
- Only binary classification tested: only binary monitoring tasks were evaluated; the accuracy of the observer's free-form answers was not evaluated (Section 6.1).
- Open questions: whether the observer is truly introspecting or only modeling assistant behavior; the authors suggest studying this with interpretability experiments or experiments that directly manipulate internal states (Section 6); building an observer channel into original post-training (training simultaneously, shaping the observer prior during mid-training) is an unexplored direction (Section 6); the assistant's misalignment tendencies may spill over to the observer, and the authors suggest not using observer reports as a training signal for the assistant (Section 6).
Scope of experiments
- The model under test is one: Qwen3.6-27B; the cross-model ablation used two external models, Inkling-Small and Nemotron 3 Super (Appendix H).
- The oversight evaluation covers 5 task types and 6 evaluation sets, with sample sizes ranging from 40 to 1500 (Table 1); there is also a behavior prediction evaluation with 489 questions (Table 2).
- Baselines cover four types: follow-up, self/external transcript monitoring, honesty training, and activation oracle; no comparison with chain-of-thought monitoring (Section 6.1).
- Capability retention evaluation covers MMLU-Pro, GPQA Diamond, AIME 2025/2026, and IFEval (Table 3).
- The paper does not report the number of repeated runs for the oversight evaluation; the Figure 4 caption notes that error bars come from a bootstrap over the 1,500-question sycophancy set, not from the 400-question confidence interval.
Summarize the main content of the paper
Training an observer role that shares weights with the assistant to answer verifiable introspection questions achieves an average AUROC of 0.907, exceeding external monitoring, honesty training, and the activation oracle.
A supervision methods paper: for behaviors that are difficult to detect in transcripts, such as reward hacking, deception and sycophancy, it proposes introspective oversight—training a third-person observer role that shares weights with the assistant on Qwen3.6-27B to answer arbitrary questions about the assistant's behavior, with answers verified against the assistant's actual sampled behavior.
- Method: In the SFT stage, two types of data—behavior prediction and hidden-trait (24,950 each)—are used to establish a question-answering channel; in the RL stage, counterfactual prediction (Brier score reward) and multiple-choice prediction (distractors drawn from four other models) teach the observer to reason; on-policy distillation is used throughout to prevent assistant drift, with rank-32 LoRA.
- Main results: On five held-out oversight evaluations, the observer achieves an average non-reasoning AUROC of 0.907, higher than honesty training (0.864), external transcript monitoring GLM-5.3 (0.828), activation oracle (0.796) and direct follow-up (0.776) (Figure 3).
- Key ablations: External models trained on the same data, Inkling-Small and Nemotron 3 Super, reach only 0.684 and 0.647, from which the authors conclude that privileged access through shared weights is the source of the advantage; the third-person observer (0.907) outperforms directly training the assistant role (0.877).
- Reasoning mode: RL raises reasoning AUROC from 0.719 to 0.823 and reasoning length from 600 to 1,700 tokens, but it remains below the non-reasoning 0.907; the authors speculate that answer format is the main cause.
- Incidental finding: After observer training, the assistant itself becomes easier to monitor—follow-up rises from 0.776 to 0.892, and every baseline such as activation oracle improves on average; the authors give three possible explanations and state that the experiments cannot distinguish among them.
- Authors' conclusion: Training models on verifiable introspection questions is a viable path to supervising unintended behavior and can complement activation, transcript and CoT monitoring; limitations include a single model, no comparison with CoT monitoring, some small evaluation sets, no testing of hidden-goal models, and testing only binary classification.