Monitorability of natural language autoencoders (NLAs): identification and discrimination

1. 01What problem does this paper attempt to solve?

   The paper explores whether natural language autoencoders in practical supervision can distinguish true and false statements about activations based on human-readable explanations.

   Core question: whether the readable explanations learned by natural language autoencoders (NLAs) under generative autoencoding training, when deployed as supervision tools, have the ability to distinguish true from false statements about activations.

   • Scenario and importance: Large language models acting as agents may exhibit misaligned behaviors such as scheming, sandbagging, or reward hacking, requiring monitoring tools for assurance. NLAs convert activation vectors into human-readable text via activation verbalizers (AVs), then reconstruct the vectors via activation reconstructors (ARs); their supervisory value depends on whether humans can use them to identify the truth of activations.
   • Inadequacy of existing training objectives: The authors state that existing NLA training objectives are entirely generative, with the loss computed only on cosine distortion for a single target activation, and never introducing negative samples or distractors to require the explanation to rule out any candidate statements.
   • Core theoretical insight: The authors formalize NLA training and supervised deployment as two different reference games in emergent communication: training is a generative game, while supervision is a discriminative, multimodal, partially observable game. From this, they conjecture that generative objectives can only enable explanations to identify activations, not to discriminate between closely detailed true and false statements.
   • Main contributions of the paper: Use the NLA's own AR to build a proxy monitoring model in the theoretically upper-bound sense, evaluate on the real WildChat conversation dataset the NLA's ability to distinguish different activation statements and single-assertion false statements, and verify whether private communication conventions exist.

   Collapse this answer

2. 02What related research exists?

   The paper reviews four research lines: emergent communication referential games, lie detection probes, chain-of-thought monitoring, and activation verbalizers.

   Deception and lie detector research

   • Goldowsky-Dill et al. (2025): Train linear probes on contrastive data to separate deceptive from honest answers in Llama-3.3-70B, with AUROC between 0.96 and 0.999; but the authors note the probes partially respond to the scenario about deception rather than the deceptive behavior itself.
   • Parrack et al. (2025) and Kretschmar et al. (2025): The former evaluates probes via black-to-white boost and finds negative gains on datasets where truthfulness is already determinable from public tokens; the latter finds in large-scale evaluation that detectors fail most when the transcript itself cannot reveal the lie.
   • Natarajan et al. (2026) and Cooney et al. (2026): The former reports that targeted probes perform better on specific deception types; the latter notes that many model organisms used for ground-truth labels do not verify that they believe the opposite of what they say.

   Chain of thought and agent monitoring

   • MonitorBench (Wang et al., 2026): Injects decision-critical factors into prompts and tests whether chain of thought reveals these factors, finding that models tend to omit rather than encode them.
   • Agent sabotage and sandbox monitoring (Ward et al., 2025; Libon et al., 2026; Li et al., 2025; Sinha et al., 2026): Evaluate sabotage and sandbagging in automated AI R&D, reporting that models can covertly sandbag to bypass chain-of-thought monitoring in 16% to 36% of successful attempts.

   Auditing model organisms and activation verbalizers

   • AuditBench (Sheshadri et al., 2026) and Pando (Zhong et al., 2026): The former finds on 56 organisms and 14 hidden behaviors that scaffolding black-box tools outperform white-box tools; the latter proposes eliciting confounders and controlling them by training models to faithfully explain, not explain, or follow distraction rules.
   • Activation oracles and verbalizers (Karvonen et al., 2026; Chen et al., 2026; Li et al., 2026): Build question-answering or deception queries against activations; Li et al. report that existing verbalizer benchmarks can be solved from input text alone, and verbalizers often report their own parametric knowledge rather than the target model's knowledge.

   Baseline methods and benchmark datasets

   • Evaluation benchmarks and baselines: Use the open-source NLA checkpoint released by Anthropic (2026) as the evaluation target, and construct comparison baselines on real WildChat (Zhao et al., 2024) conversation logs with multiple families of mechanical substitutions (top-k, top-p, donor words, etc.) and external model rewrites.

   Positioning of the paper relative to existing work

   • The authors state that existing work mainly relies on binary deception scores or agent success rates, whereas this paper separates identification from discrimination, evaluates the tool itself on rows of false statements with strictly matched specificity, and provides a formal framework indicating where white-box monitoring can yield gains.

   Collapse this answer

3. 03How does the paper address this problem?

   The paper constructs a dual-game theoretical framework, uses an activation reconstructor to build a threshold-free proxy monitor, and designs multiple classes of controlled false statements.

   The paper formalizes NLA training and supervised deployment as two types of referential games, builds a threshold-free proxy monitoring model, and designs specificity-matched control statements to evaluate discriminative ability.

   Referential game theoretical framework and core formalization

   • Formalization of the training scenario (Setting 1): Formalized as a generative referential game, where the verbalizer AV is the speaker and the reconstructor AR is the listener, with the optimization objective of minimizing cosine distortion between target activations and reconstructed activations:

   J^gen(Sp, Li_gen) = 𝔼_{s ∼ 𝒟} 𝔼_{z ∼ Sp(· | s)} [dist(s, Li_gen(z))] where dist(h, ĥ) = 1 − cos(h, ĥ). This objective never requires the generated explanation to exclude any non-target activations or candidate statements.

   • Formalization of supervised deployment (Setting 2): Formalized as a discriminative, multimodal, partially observable referential game, where the listener (monitor) cannot directly observe internal activation vectors and must score and rank the true statement against distractor statements in a candidate set.
   • Hierarchical division of truth relations: Defines the forward-propagation truth set Aproc, the transcript assertion set Ainput, and the activation assertion set Ah; the theory states that white-box monitors with transcript access can only outperform black-box monitors on Ax \ Ainput (processing-level assertions).

   Proxy monitoring model and threshold-free lineup evaluation

   • Constructing a theoretical-upper-bound proxy listener: Because human monitoring cannot be tested at scale, the paper uses NLA's built-in AR to build a proxy monitor:

   Li_AR(c; h) = cos(AR(c), h) This proxy has an oracle-augmented property (directly comparing against the hidden vector h) and a maximum-sympathy property (sharing AV's training conventions). The authors argue that its evaluation constitutes a theoretical upper bound on real monitoring performance.

   • Lineup AUC metric: For a lineup containing 1 true statement and K distractor statements, computes the average binary win rate of the true statement's score exceeding the distractors, avoiding bias from choosing a hard classification threshold, and maintaining a 0.500 random baseline for any candidate count K.

   Construction of adversarial statements and control distractors

   • Semantically rewritten statements (testing truth weighting): An independent model Qwen2.5-7B-Instruct rewrites the true explanation into adversarial statements tampering with 1, 2, or 4 specific assertions, keeping grammar, register, length, and structure consistent (word change rate only 0.018 to 0.036, 96% to 98% original tokens).
   • Surface rewritten statements (testing form sensitivity): External models perform light (lexical only) and heavy (lexical plus syntactic) paraphrases under fixed semantics.
   • Mechanical replacement baseline (semantically undefined control): Under the same number of word changes, uses AV's own top-k, top-p (0.9) sampled words, or cross-dialogue donor words (with part-of-speech matching, non-occurrence within the dialogue, and embedding distance lower bound constraints) to replace content words.
   • Word-order and heterologous control groups: Includes a shuffled group that fully preserves vocabulary but scrambles word order, and a heterologous group of true statements from other dialogues (setting the relative displacement damage baseline to 1.0).

   Quality control and evaluation mechanisms

   • Placeholder protection and semantic judge: Before rewriting, quoted text in the original explanation is masked as placeholders and restored after generation; a benchmark-validated LLM judge is introduced to confirm whether statement semantics changed (the judge's confirmation rates on single, double, and quadruple assertions are 48%, 71%, and 75% respectively), and the main experiment uses only adversarial statements whose semantic change was confirmed by the judge.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Experiments find that NLA explanations can identify activation sources, but the AUC for discriminating single-assertion false statements is only 60.2%, and reconstruction mainly responds to surface-level word changes.

   Experimental Setup

   • Models and checkpoints under test: We evaluate the NLA checkpoint open-sourced by Anthropic (2026); the AV uses a truncated Qwen2.5-7B as its backbone network and is equipped with a linear readout layer, and the AR has 21 layers (d = 3584); the activations being explained are extracted from layer 21 of the target model Qwen2.5-7B-Instruct, at the position of the 2nd-to-last token at the end of the dialogue.
   • Dataset: Dialogue samples come from human-machine interaction logs in the WildChat dataset.
   • Auxiliary rewriting model: Qwen2.5-7B-Instruct, trained independently of NLA self-play, performs paraphrase rewriting and false-statement construction.
   • Metric definitions: We use the area under the lineup curve (Lineup AUC, random baseline 0.500); supplemented by Top-1 accuracy (baseline 0.111), 40-way retrieval accuracy (40-way retrieval, baseline 0.025), and angular displacement damage.
   • Evaluation method: A validated LLM judge is used to verify the semantic change of adversarial statements (97.5% agreement rate on identical text, 100% difference rate on heavily corrupted text).
   • Sample size and environment: The adversarial statement experiment uses 40 dialogues (valid single-assertion samples n = 35), and the paraphrase experiment uses 60 dialogues; the lineup contains 8 distractors; run on a single AMD Radeon 8060S GPU (ROCm environment).

   Main Results

   According to Figure 4 and Table 1, the main experiment measured the performance of the agent monitoring model under different distractors as follows:

   Swipe horizontally to read the table

   Distractor type	Corresponding assertion/modified word count	Sample size (n)	Lineup AUC (mean)	95% confidence interval / standard error	Source
   Other dialogue activation statements	Full statement replacement	40	1.000	Not reported	Figure 4
   Judge-confirmed adversarial statements	1 assertion	35	0.602	SE = 0.067 [0.471, 0.728]	Figure 4 / Table 1
   Judge-confirmed adversarial statements	2 assertions	39	0.585	Not reported	Figure 4
   Judge-confirmed adversarial statements	4 assertions	40	0.605	Not reported	Figure 4
   Non-judge adversarial statements	1 assertion	39	0.564	Not reported	Figure 4
   Mechanical replacement control	1 word	39	0.418	Not reported	Figure 4
   Mechanical replacement control	2 words	39	0.527	Not reported	Figure 4
   Mechanical replacement control	4 words	40	0.501	Not reported	Figure 4

   • A significant disconnect between identification and discrimination: The authors point out that NLA can perfectly identify the source of activations; the true statement wins in all 320 comparisons against statements of activations from other sources, and the 40-way retrieval accuracy reaches 0.950; but when discriminating fluent false statements that tamper with only a single assertion, the win rate is only 60.2%, and the confidence interval touches the chance level of 0.500.
   • Adversarial statements are not treated as noise: In the 40-way activation retrieval experiment, replacing the original statement with an adversarial statement still retrieves the corresponding activation with a success rate of 0.949 (0.950 for the original statement), indicating that the activation traceability of the text is almost undamaged after tampering with the assertion, and the discrimination failure is not because the text is corrupted and the reconstructor cannot read it.
   • Reconstruction closely tracks surface word form rather than factual truth: A single-word tampering deflects the reconstructed vector by 1.6 degrees (far below the AR's own inherent reconstruction error of 22 degrees), and the deflection direction is nearly perpendicular to the inherent error (87.8 degrees), so that the reconstruction's error relative to the true activation increases by only 0.06 degrees, causing the reconstructor's response to factual changes to be masked by surface word changes.

   Private Communication Dialect Test (Experiment #2)

   • What was tested: Whether the AR forms a private dialect due to self-play training, causing it to give low scores because the statements come from an external model rather than because the assertions are false.
   • Experimental results: The median reconstruction displacement caused by semantics-preserving rewriting by the external model accounts for only 0.021 of the distance between different activations (Figure 7), and the 40-way retrieval accuracy remains 0.917 before and after rewriting; in contrast, the displacement caused by shuffling word order reaches 0.152, 7 times the former; the damage difference between external paraphrase and internal top-k replacement under matched word-change degree is +0.073 ± 0.034 (p = 0.041).
   • Authors' interpretation: The authors state that the AR can understand English written by external models almost losslessly, and its private conventions are mainly confined to word order rather than vocabulary itself; while adversarial statements strictly preserve the original word order and 96% to 98% use the AV's original tokens, so the discrimination failure does not stem from the unfamiliarity of external language, but from the inability to assign weight to truthfulness.

   Word Replacement Benchmark and Reconstruction Geometry Mechanism Analysis

   • What was tested: The win rate of 6 types of mechanical word replacement control groups under single-word modification, and the geometric evolution of perturbations in activation space (Figure 5 and Table 1).
   • Experimental results: As shown in Table 1, the exclusion rates of the 6 types of mechanical constructions under 1-word modification are all between 0.480 and 0.509, strictly near the chance level; only the false statements that change semantics reach a win rate of 0.602; Figure 5 shows that the reconstruction displacement caused by a single-token edit is only 1.6 degrees, and the power-law fit indicates that about 62 tokens need to be tampered with to reach the AR's inherent 22-degree reconstruction error level.
   • Authors' interpretation: The authors state that the weak truthfulness signal of false statements has a real directionality (win rate higher than mechanical replacement), but due to the AR's Lipschitz smoothness, an extremely small surface word-form change can only cause an extremely weak reconstruction displacement, thereby geometrically locking in the inevitability that the generative autoencoder cannot discriminate local facts.

   Other Ablations and Analyses

   • The exclusion rate of top-k word replacement is 0.509 (SE 0.016, Table 1).
   • The exclusion rate of top-p 0.9 word replacement is 0.492 (SE 0.017, Table 1).
   • The exclusion rate of single-character replacement with donor words matched for part of speech and not constrained in the input is 0.480 (SE 0.025, Table 1).
   • 5% of semantics-preserving rewrites have a displacement loss exceeding 0.25 of the distance between different activations (Figure 7).
   • A low-rank bilinear probe on frozen AR outputs distinguishes surface-rewritten statements with an AUC of 0.903, and the AR cosine readout is 0.533 (Section 9).

   Collapse this answer

5. 05What are some points that can be further explored?

   The paper notes limitations such as not testing processing-level assertions and not sampling free hallucinations, and the experimental coverage is limited to a single checkpoint and transcript-level assertions.

   Limitations and future directions noted by the authors

   • Sample size and unfilled rows of the matrix (Section 7): The experiments used a single random seed throughout; the adversarial paraphrase experiments covered only 40 dialogues; the matrix had 119, 58 and 40 unfilled slots for single-assertion, double-assertion and four-assertion respectively; and only 48% of single-assertion candidates were confirmed by the judge to have changed the semantics.
   • Limited to transcript-level assertions (Section 7): All adversarial paraphrases tested only assertions verifiable from the transcript (Ainput), and did not test internal-processing-level assertions such as model intent and planning, which supervision focuses on most (Ah \ Ainput).
   • Shared pretraining backbone (Section 7): The rewriting model Qwen2.5-7B-Instruct shares pretrained weights with the AV backbone, so implicit linguistic conventions inherited from the pretraining stage were not ruled out.
   • Free-sampling confabulation not tested (Section 7): All test texts were manual or external edits of AV outputs, and completely did not cover confabulated text that the AV autonomously generates when free sampling with temperature.
   • Introducing a discriminative training objective (Section 9): The authors propose adding a multimodal discriminative term to the NLA training objective, and exploring whether fine-tuning on the existing checkpoint can improve discrimination of local false statements.
   • Building a benchmark for causal intervention on internal states (Section 9): The authors suggest inducing known internal states by means such as manipulating role-vector activations, stripping the system prompt, or fine-tuning the model organism, and developing a multi-activation monitoring architecture that interleaves text and activation inputs.

   Scope of experimental coverage

   • Checkpoint coverage tested: Only 1 publicly released NLA checkpoint from Anthropic was evaluated (AV based on truncated Qwen2.5-7B, AR with 21 layers and d = 3584).
   • Activation extraction location coverage: Only one activation vector was tested, at layer 21 of Qwen2.5-7B-Instruct, at the second-to-last token position at the end of the dialogue.
   • Data and rewriting coverage: All evaluation dialogues were taken from human-machine interaction samples in the WildChat dataset, and rewriting and adversarial generation were performed only by the single model Qwen2.5-7B-Instruct.
   • No transcript baseline comparison reported: The paper did not run a black-box monitoring control experiment that directly reads the transcript, did not test activations at multiple layers or multiple token positions, and did not report manual evaluation results from human monitors.

   Collapse this answer

6. 06Summarize the main content of the paper

   The paper finds that although NLA explanations can identify activations, they struggle to distinguish local false statements, pointing out the boundary of generative autoencoder training for safety supervision.

   • Paper positioning: This study is an analytical paper on interpretability and monitoring tools for large language model internal states, examining the validity boundaries of natural language autoencoders (NLA) from the perspective of emergent communication theory.
   • Core question: NLA generates readable text explanations of model activations to aid safety alignment, but its training only optimizes single-vector generative reconstruction and never learns to exclude adversarial distractors, facing the fundamental challenge of questionable ability to discriminate true from false statements in supervision scenarios.
   • Method highlights: The authors formalize training and supervision as generative and discriminative referential games respectively, use NLA's own activation reconstructor to build a theoretical-upper-bound-level proxy monitor, and construct high-fidelity false statements that change only a single assertion for threshold-free lineup evaluation.
   • Key experimental results: On WildChat real dialogue tests, the proxy monitor achieves a 100% win rate in heterologous activation identification (winning all 320 comparisons), but only a 60.2% AUC in distinguishing single-assertion false statements (n=35, confidence interval touching the 50% chance level); semantically preserving rewrites have a median displacement of only 2.1% of the heterologous activation distance, ruling out the private-dialect explanation; single-assertion tampering increases reconstruction error by only 0.06 degrees, with reconstruction overly constrained by surface word-form changes.
   • Conclusion and implications: The authors argue that activation identification and truth discrimination are mechanistically decoupled, and autoencoder reconstruction does not guarantee that the model has factual supervision value; future NLA construction must introduce discriminative loss terms in training and force evaluation against local false statements with equal specificity.

   Collapse this answer

Related papers