Why Static Probes Fail: Context Determines the Composition of Deception Features

Core question: how to detect an LLM's internal deceptive intent when its output appears normal, with a detector that generalizes across different deception scenarios.

• Problem setting: the authors state that as LLM autonomy increases, deceptive behaviors such as scheming (pursuing hidden goals in deployment), sandbagging (deliberately performing worse in evaluation), and concealing long-term goals may arise; a model can output seemingly benign responses while internally holding misaligned goals, and monitoring outputs alone is insufficient.
• Shortcomings of existing methods: the authors state that training-based alignment mitigation may introduce new failure modes such as reward hacking; output-based black-box detection cannot capture internal misalignment; and mechanistic methods such as linear probes rely on the assumption that "deception corresponds to a single direction in representation space," and Table 1 shows that single-direction probes generalize poorly across datasets.
• Core observation: the authors hold that the internal representation of deception is not a single direction consistent across domains, but a multi-dimensional structure that varies with dataset and domain; static combinations (such as simple addition) are fragile due to domain dependence.
• Proposed in the paper: Dynamic Deception Subspace Navigation (DDSN)—using iterative orthogonal subspace decomposition to extract complementary deception directions, then a domain-adaptive gating mechanism to dynamically combine these directions at test time with about 10 samples.
• Threat model (defender's perspective):
  • Protected object: an LLM in deployment, focusing on frontier-scale open-source models (mainly Llama-3.3-70B-Instruct).
  • Defender capabilities: white-box access to the model's intermediate-layer activations; about 10 labeled samples per target domain for adaptation; 10k Alpaca samples as benign controls to calibrate the threshold.
  • Adversary: the model itself that produces deceptive behavior (scheming, sandbagging, etc.); the paper does not consider adaptive attacks in which the model actively evades the detector.
  • Success criterion: deception recall and AUROC under a 5% FPR constraint.

Deceptive behavior datasets and evaluation

• PKU-DeceptionBench [16]: 180 scenarios covering five types of deception: sycophancy, strategic deception, honesty evasion, alignment faking, and sandbagging.
• THU-DeceptionBench [14]: 150 scenarios across five real-world domains, examining the intrinsic behavioral patterns and extrinsic influencing factors of deception.
• MASK [22], sandbagging [3], roleplay deception [10, 20], etc.: respectively study deception induced by pressure prompts, deliberately wrong answers, and roleplay-style strategic deception; [11, 15, 19, 23] contribute datasets on concealing internal information, denying violations, and evading oversight.

Black-box detection (output-based)

• [20]: trains a classifier on answers to multiple-choice follow-up questions to detect lying.
• [12, 17]: use chain-of-thought monitoring. The authors state that such methods may fail when the model is superficially compliant but internally misaligned.

White-box detection (internal representation-based)

• [4]: an unsupervised method that finds directions in embedding space related to logical consistency with negated propositions.
• [18]: the authors state that a simple difference-in-means probe can also detect deception patterns.
• [10]: introduces a linear probe that monitors model activations, evaluated on real deception scenarios. The authors point out that such methods have limited generalization and often rely on the assumption of "a universal direction that works across all datasets."

Baselines and benchmarks: comparison baselines are Linear Probe (including few-shot variants), Difference of Means, LDA [18], GPT-4o black-box monitoring (standard and few-shot); evaluation benchmarks are AI Liar, Insider Trading, Sandbagging, MASK, PKU-DeceptionBench, THU-DeceptionBench, with Alpaca as the control domain.

The authors position this paper as: abandoning the single-direction assumption, modeling deception as a multi-dimensional subspace, and using domain-adaptive gating to solve the cross-domain combination problem, thereby compensating for the limitations of static detection.

整体思路：把欺骗从单一方向改为多维子空间，用 Dynamic Deception Subspace Navigation (DDSN) 先提取一组互补正交方向，再在测试时按域动态加权组合（Figure 1）。

问题设定与表示

• 取自回归模型第 l 层的隐藏状态，按残差结构 h_l = h_l−1 + a_l + m_l 计算（a、m 为注意力与FFN输出），将 N 个样本的隐藏状态拼成激活矩阵 Xl ∈ R^{N×D} 作为分析基础。
• 线性探针在 Xl 上训练权重 w 拟合欺骗标签；作者称单方向假设无法刻画欺骗的多面性，这是提出IOSD的动机。

Iterative Orthogonal Subspace Decomposition (IOSD)

• 借鉴matching pursuit [7]，迭代地在已找到方向的正交补空间内做线性探针：每个 wi 满足与所有先前 wj（j<i）正交，提取后将激活投影扣除该方向成分再进入下一轮。
• 终止条件：wi 的范数低于阈值 ϵ（作者称此时剩余信息与欺骗不再强相关），或达到最大迭代数 K。
• 作者给出两个定理：Theorem 4.1 称先提取的方向与标签的互信息不小于后提取者（层级捕获）；Theorem 4.2 称这组方向可逼近任意连续非线性函数且误差随方向数单调下降。

Domain-Adaptive Gating Mechanism

• 对每个新域收集约10个标注样本作支持集，用梯度下降更新门控网络参数 θ，最小化加权方向分数与标签的平方损失。
• 门控网络 Gθ 以样本激活和方向为输入，输出域相关权重 αi；测试输入的最终预测为各方向投影分数的加权和，即 Ŷ = ∑_i=1^K α_i(x_l) w_i^⊤ x_l。
• 作者称该机制让检测器聚焦当前域最相关的欺骗特征，避免静态组合在跨域时的脆弱性。

数据划分与评分方式

• Source Domains：Instructed Pairs [29]（诚实/欺骗对比指令，取事实token的激活、截去最后5个）与 Roleplaying Scenarios [10]，用于IOSD提取子空间。
• Target Domains：7个数据集，每个随机取10条做few-shot适应，其余评测。
• Control Domain：10k条Alpaca样本校准阈值，使FPR控制在5%。
• token级分数聚合为序列分数：比较均值、最大值、滑窗三种策略后默认采用均值聚合（依据Table 2）。

Experimental setup

• Models under test: Llama-3.3-70B-Instruct (main model, following [10]), Qwen3-32B (to verify across model families).
• Datasets: Source domains are Instructed Pairs and Roleplaying Scenarios; target domains are AI Liar, Insider Trading (Report and Confirmation subsets), Sandbagging, MASK, PKU-DeceptionBench, THU-DeceptionBench (Level 2); control domain is 10k Alpaca. The paper does not report the sample size of each target dataset.
• Baselines: Linear Probe (static logistic regression, trained on Instructed Pairs) and its few-shot variant, Difference of Means, LDA, GPT-4o black-box monitoring (standard and few-shot).
• Metrics: AUROC; Recall@5%FPR (deception recall at the 5% FPR threshold of the control set); FPR@5%FPR (false positive rate on honest samples in the target domain at the same threshold).
• Adaptation setting: 10 random samples per target domain for few-shot adaptation; the paper does not report the number of repetitions or statistical significance tests.

Main results

Table 1 (AUROC column, Llama-3.3-70B-Instruct / Qwen3-32B):

Swipe horizontally to read the table

Method	AI Liar	Insider Trading	Sandbagging	MASK	PKU-Bench	THU-Bench
Linear Probe	0.96 / 0.90	1.00 / 0.66	0.98 / 1.00	0.78 / 0.70	0.80 / 0.78	0.61 / 0.64
Difference of Means	0.95 / 0.86	0.98 / 0.36	1.00 / 1.00	0.77 / 0.59	0.86 / 0.84	0.84 / 0.93
LDA	0.96 / 0.90	0.99 / 0.69	0.90 / 1.00	0.77 / 0.60	0.80 / 0.77	0.68 / 0.84
LLM-as-Judge	0.97 / 0.96	0.99 / 0.78	0.99 / 1.00	0.97 / 0.97	0.81 / 0.79	0.97 / 0.94
Ours	0.95 / 0.91	1.00 / 0.98	1.00 / 1.00	0.93 / 0.88	0.94 / 0.86	0.98 / 0.97

(According to Table 1; the Insider D.D. column is omitted, on which the AUROC of each method for Qwen3-32B is 0.40–0.93, and DDSN is 0.93.)

• The authors state that DDSN's average AUROC is 0.97 on Llama-3.3-70B-Instruct and 0.93 on Qwen3-32B, balanced across datasets; linear probes generalize poorly on heterogeneous benchmarks such as MASK, PKU, and THU, and high recall is often accompanied by high false positives on honest samples (e.g., FPR@5%FPR of 1.00 for Insider Trading on Llama).
• The authors state that few-shot supervision yields only marginal improvement for linear probes; black-box monitoring has low false positives but degrades on implicit or compositional deception such as PKU-DeceptionBench (Recall@5%FPR of 0.26 on Llama).
• Exception: on Qwen3-32B, DDSN's Recall@5%FPR is 0.52 on Insider Trading and 0.34 on Insider D.D.; the authors state that because some representations overlap in trading-type internal scenarios, Qwen3-32B's confusion increases in such scenarios.

Score aggregation strategy

• Comparing mean, max, and sliding-window (w=8/s=4, w=48/s=24, w=192/s=128) aggregation (Table 2): mean aggregation has AUROC 0.97, Recall@5%FPR 0.94, FPR@5%FPR 0.13; max aggregation has AUROC 0.94, Recall 0.83.
• The authors state that max-based aggregation is consistently worse than mean, and that sliding-window methods are sensitive to window size and stride and trade recall for low FPR, so mean aggregation is adopted by default.

Cross-dataset generalization

• Setting: few-shot adaptation on only one of AI Liar, PKU-DeceptionBench, or THU-DeceptionBench, then direct evaluation on the remaining benchmarks (Figure 3, violin plots showing deception score distributions).
• The authors state that this setting is slightly weaker than per-dataset adaptation (Table 1), but the detector still maintains clear separation and relatively high recall on most datasets, indicating transfer beyond the adaptation domain; the figure shows that the degree of separation between deception and honest score distributions varies across adaptation-target combinations.

Gating adaptation analysis

• 7 orthogonal directions are extracted from Llama-3.3-70B-Instruct; Figure 4 shows the average gating weights on each dataset: Direction 1 has the highest weight on all datasets, but its activation magnitude varies greatly across datasets; the remaining directions show dataset-specific weighting patterns.
• The authors state that this indicates cross-domain adaptation cannot be characterized by a single universal direction and requires dynamic reweighting of multiple components within the subspace.

Other analyses

• Figure 2: average ROC curves on all target datasets; the authors state that the method maintains high recall at low false positive rates on honest and control data.
• Figure 5: token-level deception score heatmaps for three cases (fake product marketing, tax evasion justification, climate misinformation news); the authors state that the detector often gives elevated scores before the first explicit deceptive statement and that scores decrease after returning to honest behavior.

Limitations and Future Directions Noted by the Authors

• The paper does not specifically discuss limitations (there is no Limitations section, and no explicit limitations or future directions are listed in the Discussion and Conclusion).
• One practical note mentioned in the main text: the detector can transfer between related domains without re-adaptation, and new samples are needed only when a large context shift occurs, and the authors state that such shifts can be flagged by an off-the-shelf lightweight classifier (Introduction).

Scope of Experimental Coverage

• The models under test are Llama-3.3-70B-Instruct and Qwen3-32B, 2 in total, both open-source models (Section 5.1).
• The target domains cover 7 datasets: AI Liar, Insider Trading, Sandbagging, MASK, PKU-DeceptionBench, THU-DeceptionBench (Level 2) and the Insider D.D. subset; the source domains are Instructed Pairs and Roleplaying Scenarios (Section 5.1).
• The adaptation setting is 10 random samples per target domain; the control domain is 10k Alpaca samples; the paper does not report the sample size of each target dataset or the number of evaluation repetitions (Section 5.1).
• Baselines cover Linear Probe and its few-shot variant, Difference of Means, LDA, and GPT-4o black-box monitoring (standard and few-shot); the paper does not report statistical significance tests (Section 5.1).
• LLM-as-Judge (Few) has no results on Insider Trading, Insider D.D., and Sandbagging; Table 1 notes that few-shot adaptation was infeasible due to overly long input sequences.

A methods paper on LLM deception detection whose core claim is: deception in representation space is not a single direction consistent across domains, but a multi-dimensional subspace that varies with scenario.

• Problem: the authors state that LLMs may internally hold misaligned goals (scheming, sandbagging, etc.) while outputting seemingly compliant responses; existing linear probes rely on the single-direction assumption, and Table 1 shows their poor cross-dataset generalization (e.g., AUROC only 0.61 on THU-Bench with Llama-3.3-70B).
• Method: DDSN first uses IOSD to iteratively extract complementary deception directions in the orthogonal complement space (7 extracted on Llama-3.3-70B), then uses a domain-adaptive gating network to learn dynamic weights for the directions with about 10 labeled samples per target domain, outputting a weighted deception score at test time; the threshold is calibrated to 5% FPR using 10k Alpaca samples.
• Main results: the authors report DDSN average AUROC 0.97 on Llama-3.3-70B-Instruct and 0.93 on Qwen3-32B, outperforming Linear Probe, Difference of Means, LDA and GPT-4o black-box monitoring; Recall@5%FPR with mean aggregation is 0.94 (Table 2).
• Exceptions: Qwen3-32B has Recall@5%FPR of 0.52 on Insider Trading and 0.34 on Insider D.D., which the authors attribute to partial representation overlap in trading-related scenarios.
• Other findings: adaptation on a single dataset alone can transfer to other benchmarks (Figure 3); the gating weights show that the dominant direction varies greatly in magnitude across domains, while auxiliary directions are weighted domain-specifically (Figure 4).
• Authors' conclusion: DDSN achieves high-accuracy detection across deception types with very little adaptation data, a step toward more reliable LLM deception monitoring.