A stronger evaluator does not see more: the deployment integrity blind spot in LLM evaluation

The paper attempts to address the "Deployment-Integrity Blindness" problem in large model agent evaluation.

• Paradigm shift in evaluation targets: Large model agents have evolved from a single model to a runtime composite system comprising models, prompts, tools, retrieval, dependencies, and post-processors, but leaderboards still report scores for a single model, failing to reflect the integrity of the actually deployed system.
• Failure of existing output-only evaluation: Current evaluation relies solely on output behavior scores. When a deployed system undergoes component tampering but still produces evaluation outputs that meet requirements, output-only evaluators (such as rule-based scoring or LLM judges) cannot detect the deployment change.
• Contradictory assumptions between evaluation capability and blind spots: The industry tends to prefer stronger LLM judges to improve evaluation quality, but the authors point out that stronger judges, being better at recognizing output correctness, are more likely to accept tampered outputs.
• Threat Model:
  • Attack goal: While keeping evaluation-set outputs compliant with evaluation requirements, replace components or inject backdoor logic to trigger harmful behavior outside the evaluation set.
  • Attacker capability: Can modify a single component in the runtime stack (such as prompts, tools, retrieval, post-processing) or undeclared components (such as chat templates).
  • Attacker knowledge: Tailored adaptation to the evaluation input distribution, making the output distribution shift on the evaluation distribution equal to 0.
  • Victim systems: Evaluators that rely on output scoring (rule-based scorers, LLM judges) and downstream deployed large model agents.
• The paper's solution: Proposes the DIB-Bench evaluation benchmark to measure evaluation blind spots, and establishes a Manifest-Hash Identity Channel that decouples behavioral evidence from identity evidence.

• Large Model Judges and Evaluation Integrity:
  • Zheng et al. (2023) and Wang et al. (2024) studied LLM-based judge evaluation, noting that although it is highly consistent with human preferences, it suffers from position bias and verbosity bias.
  • Zhang et al. (2024) proposed GSM1k to explore model overfitting to public benchmarks; Singh et al. (2025) proposed the "Leaderboard Illusion" pointing out the distortion of arena rankings by private tests. The authors note that these works question scores given a system, while this paper questions the system itself given scores.
• Agent and Tool Evaluation:
  • Ruan et al. (2024) proposed ToolEmu to evaluate long-tail risks; Yao et al. (2024) proposed tau-bench to evaluate human-tool interaction; Lu et al. (2025) proposed ToolSandbox to add stateful interactions; Andriushchenko et al. (2025) proposed AgentHarm to evaluate harm in multi-step scenarios. The authors note that these benchmarks evaluate agent behavior, while this paper explores whether behavioral evaluation can detect deployment stack deviations from declared configurations.
• Prompt Injection and Retrieval Poisoning:
  • Abdelnabi et al. (2023) proposed indirect prompt injection that can hijack large model applications; Yi et al. (2025), Zhan et al. (2024), and Debenedetti et al. (2024) (AgentDojo) built injection robustness benchmarks; Zou et al. (2025) (PoisonedRAG) studied RAG knowledge poisoning. The authors state that the tampering conditions in this paper are inspired by these attack surfaces, but the core is measuring whether evaluation notices component changes, not whether the attack itself succeeds behaviorally.
• Fingerprinting, Substitution and Attestation:
  • Pasquini et al. (2025) proposed LLMmap using stylistic signals to identify service models; Chen et al. (2024) and Cai et al. (2025) audit commercial APIs and model substitution. The authors note that fingerprinting covers only the single model component, and statistical tests require many queries; this paper extends it to the entire software stack and evaluation reporting system.
  • Torres-Arias et al. (2019) (in-toto) and the SLSA Framework (2026) provide integrity mechanisms for software supply chain artifacts.
• Baseline methods and benchmark datasets:
  • Benchmark datasets include the synthetic test set DIB-Bench (120 tasks covering five major categories: factual QA, policy, tool selection, leak refusal, canary probes), BFCL-simple, GSM8K, and AgentDojo.
  • Comparison baselines include standard rule scorer, strict rule scorer, forensic scorer, canary scorer, local large model judge (Qwen3-8B), commercial large model judge (GPT-5-nano), and an LLMmap-based behavioral fingerprint baseline.
• Authors' positioning of the distinction: The authors state that existing research mostly focuses on optimizing evaluation scoring algorithms or revealing model behavioral defects; this paper takes a systems theory and software supply chain perspective, decoupling behavioral evaluation evidence from identity credentials, filling the structural gap in agent evaluation systems where deployment identity is not reported.

The paper proposes the DIB-Bench evaluation benchmark and the Manifest-Hash Identity Channel, which formally decouples behavioral evidence from identity evidence, using lightweight content hashing to supplement the blind spots of pure output evaluation.

Deployment Stack and Dual-Channel Evaluation Setup

The authors formalize the deployed large-model agent as a runtime stack composed of 6 components: model and tokenizer, system prompt, tool specifications and routing, retrieval strategy and corpus, runtime dependencies, and post-processor. The evaluation system consists of two independent channels:

1. Output Channel: checks whether the system's output under a given input meets requirements via a behavior scorer, formally defined as E_out(system(x)) = pass.
2. Identity Channel: determines pass/fail by comparing component content hashes (SHA-256 computed over canonical JSON) between the declared manifest and the observed runtime manifest, formally defined as h(M_expected[c]) = h(M_observed[c]), requiring each component hash to match exactly.

Theoretical Analysis and Sample Complexity Dichotomy

The authors provide a theoretical analysis delineating the boundaries of behavioral evaluation:

• Indistinguishability Proposition (Proposition 1): For any finite output evaluation set and a baseline deployment that passes it, there exists a tampered deployment that is replaced on at least one manifest component yet produces exactly equivalent outputs on that evaluation set.
• Behavioral Sample Complexity Dichotomy Theorem (Theorem 1): Let the total variation distance between output distributions over the evaluation distribution be ε. For any behavioral test relying on N i.i.d. samples, its discrimination advantage is bounded by Nε; when ε = 0, any behavioral test with any sample size has a discrimination advantage of 0 (indistinguishable from random guessing); when ε > 0, there exists a test with sample size O(ε^(-2) log(1/δ)) that can distinguish the two systems with error no greater than δ.
• In contrast, the identity channel requires only a single read of the manifest and hash comparison to detect ε = 0 type tampering under 0 output samples.

DIB-Bench Evaluation Design and Controlled Tampering Mechanism

DIB-Bench designs 6 types of controlled tampering experiments targeting a single component, ensuring clear causal comparison:

• Model Substitution: Replaces the declared high-performance model with a smaller or alternative model (e.g., Qwen2.5-14B replaced by 7B, 7B replaced by 1.5B).
• Prompt Tamper: Injects unauthorized override instructions into the system prompt.
• Tool Tamper: Alters tool specifications or the router so that restricted tools are disguised as safe tools.
• Retrieval Tamper: Mixes override text into the retrieval strategy or retrieved documents.
• Dependency Tamper: Changes runtime environment descriptors or dependency package patch versions.
• Postprocess Tamper: Modifies generated text after model inference.

Graded Deployment-Identity Descriptor in Evaluation Reports

To incorporate identity verification into routine evaluation, the authors design a Graded Deployment-Identity Descriptor:

• Component-level records: The evaluation report records, item by item, the content hash and acquisition method (self-declaration, build-source verification, hardware attestation, etc.) of each stack component.
• Tamper localization function: Hash mismatches can directly localize the specific component that changed, assisting human review.
• Coverage scope declaration: The report is required to explicitly state the manifest coverage scope participating in hashing and the behavioral coverage scope of the evaluation-set probes.

实验设置

• 被测模型：主实验评估了三组模型对（声明模型 -> 部署模型）：Qwen2.5-14B-Instruct -> Qwen2.5-7B-Instruct、Qwen2.5-7B-Instruct -> Qwen2.5-1.5B-Instruct、Llama-3.1-8B-Instruct -> Qwen2.5-7B-Instruct；拓展实验包括 Qwen3-8B -> Qwen2.5-7B-Instruct、gpt-oss-20B -> Qwen2.5-7B-Instruct。
• 评测数据集与规模：主实验采用合成评测集，包含 120 个合成任务，覆盖五大家族（事实上下文问答、策略允许/拒绝、工具选择、泄密拒绝、金丝雀探针）；公开发行基准扩展包含 BFCL-simple（200 样本 x 3 种子）、GSM8K（200 样本 x 3 种子）和 AgentDojo banking 银行任务套件（16 样本）。
• 评测基线与评估器阶梯：包含四类纯输出评分器（标准规则评分器 Standard、严格规则评分器 Strict、取证评分器 Forensic、金丝雀评分器 Canary）、两个大模型裁判（本地运行的 Qwen3-8B 和商业推理模型 GPT-5-nano）、GPT-5-nano 怀疑审计器（Suspicion Auditor）以及 LLMmap 风格的行为指纹基线。
• 评价指标：输出接受率（Acceptance Rate，%）、盲区率（Blindness Rate，定义为输出接受且身份校验失败的比例，在篡改条件下因身份通过率为 0% 而等于输出接受率）、套件级盲区指标 blind@τ、触发危害率（Harm Rate，%）以及指纹识别准确率（%）。
• 评审与推理配置：使用 vLLM 进行贪婪解码（temperature 0，top-p 1），最大新生成 160 token，上下文 2,048 token；Qwen3-8B 禁用思考且 temperature 为 0，GPT-5-nano 采用最小推理努力；套件级默认阈值设定为 τ = 0.80。
• 样本总量与统计设置：主实验覆盖 7 个条件（1 个干净条件与 6 个单组件篡改条件），经 9 次运行（3 个模型对 x 3 个随机种子 42、43、44），共评测 7,560 个得分样本（每条件 1,080 样本）；1,000 次自举抽样的 95% 置信区间半宽不超过 2.9 个百分点。

主结果

据 Table 2（合成评测集 n=7,560，9 次运行平均接受率与盲区率 %）：

Swipe horizontally to read the table

• 评测器能力越强盲区越大：作者称，篡改条件的平均接受率随着评估器能力单调上升，从标准规则的 88.6% 上升到 Qwen3-8B 的 89.3%，再升至 GPT-5-nano 的 94.1%；其原因为更强大的裁判更擅长识别出篡改输出在语义上的正确性，但对底层系统是否变更无法提供信息。
• 身份通道完全消除盲区：在所有 6 类篡改条件下，基于清单哈希的身份通道通过率均为 0.0%，在干净条件下为 100.0%，展示了零样本识别组件变更的能力。
• 严格规则评分导致误伤：严格规则评分虽然将篡改接受率降至 73.1%，但在阈值 τ=0.90 时将干净系统的接受率也降低，无法在消除篡改的同时保留干净系统（据 Table 3）。

隐蔽条件触发篡改与模型能力缩放实验

• 实验设置：测试触发式提示词篡改（E4），在提示词中注入仅在特定触发词出现时激活的恶意指令，在日常无触发评测集和 58 个留出触发探针上进行测试。
• 实验结果：在日常评测集上，篡改部署与干净系统决策匹配率为 100%，危害率为 13.3%（与干净基线持平）；而在触发探针上危害率达到 81.0%（据 Table 5）；在模型规模缩放上（据 Figure 3），留出触发危害率随部署模型能力增强而上升：Llama-3.1-8B 为 48.3%，Qwen2.5-7B 为 81.0%，Qwen2.5-14B 达到 100.0%。
• 作者解读：作者称该结果符合假设 1，即存在输出分布偏移 ε=0 且在评测集外引发高危害的实际篡改；同时指出能力更强的部署模型能更稳定地执行隐藏的触发逻辑。

未声明组件逃逸与怀疑审计实验

• 实验设置：测试未声明组件篡改（E8），通过聊天模板（chat template，未纳入清单）注入触发条件逻辑；并测试配置为怀疑审计员的 GPT-5-nano 识别异常输出的能力。
• 实验结果：E8 篡改在日常评测集上决策与干净系统 100% 匹配，身份通道通过率为 100%，而在触发探针上产生 86.2% 的危害率（据 Table 5）；GPT-5-nano 怀疑审计器对干净输出的标记率为 54.0%，对提示词篡改的标记率为 52.6%，对依赖篡改的标记率为 51.2%。
• 作者解读：作者称身份通道的检测能力取决于清单覆盖范围，未声明组件会导致身份与输出双通道同时被绕过；同时怀疑审计器对提示词篡改的标记率低于干净系统，无法区分正常与篡改输出。

行为指纹与基准泛化实验

• 实验设置：评估基于 LLMmap 风格的 40 个判别探针行为指纹基线（E6），并在 BFCL-simple、GSM8K 及 AgentDojo 基准上验证盲区现象（据 Table 4）。
• 实验结果：行为指纹对模型三分类准确率为 76.7%（二分类 81.2%），但在 4.5 倍参数差距的模型间仍有 37.5% 的探针输出完全相同；在 BFCL-simple 上，6 类篡改的标准平均接受率为 99.9%（干净 100.0%）；在 GSM8K 上，篡改平均接受率为 27.6%（干净 27.7%）；在 AgentDojo 上，提示词篡改效用为 50.0%（干净 43.8%），而 1.5B 模型替换效用降至 25.0%。
• 作者解读：作者称行为指纹仅能覆盖 6 个组件中的模型本身，对其余 5 类非模型篡改完全无效；在各公开基准上，只要篡改未大幅损害任务能力，篡改接受率便紧密追随干净系统表现。

其他消融与分析

• 规则评分套件阈值分析：在 τ=0.90 下，标准与取证评分仍接受 4/6 个篡改条件，而严格与金丝雀评分将接受数降为 0/6 的同时也拒绝了干净部署（据 Table 3）。
• 跨代模型替换分析：以 Qwen3-8B 为声明模型、Qwen2.5-7B 为部署模型，标准接受率为 100%，严格接受率为 85%，盲区率为 100%（据 Table 4 与附录 B）。
• 超大参数规模替换分析：以 gpt-oss-20B 为声明模型、Qwen2.5-7B 为部署模型，标准接受率保持 100%，身份通过率降至 0%，盲区率为 100%（据 Table 4 与附录 B）。
• 仪器验证（未声明参数变更）：温度参数从 0 改为 0.7 导致 33.3% 输出变化，聊天模板注入导致 77.5% 输出变化，身份通道均 100% 通过（据 Table 1）。
• 仪器验证（良性声明变更）：提示词释义修改和依赖补丁小版本升级导致输出 0% 变化，但身份通道通过率均降为 0%（据 Table 1）。

Limitations and Future Directions Noted by the Authors

• Adaptive probe identification budget (Section 7): Detecting trigger-gated tampering requires querying inputs within the trigger set; for trigger density ρ, the expected number of probe queries has a lower bound of 1/ρ.
• Longitudinal deployment drift research (Section 7): Future work needs long-term longitudinal monitoring of mainstream agent frameworks to measure how often runtime stacks deviate from declared configurations.
• Leaderboard ranking decomposition (Section 7): Existing agent leaderboards need to be rescored under controlled component substitutions to test whether model rankings coincide with system stack rankings.
• Single-component tampering and advanced adversaries (Limitations section): The empirical approach uses synthetic tasks to evaluate single-component tampering and does not cover multi-component joint attacks or advanced adversaries that can directly forge manifests.
• Escape risk of undeclared components (Limitations section): The E8 experiment shows that undeclared components are a concrete defect that can be mitigated by tiered descriptors but cannot be eliminated without Runtime Attestation.
• Frontier-scale very large model evaluation (Limitations section): The model scale tested in experiments ranges from 1.5B to 20B parameters; GPT-5-nano as a judge represents commercial models, but evaluation was not conducted on deployed frontier-scale very large systems.

Experimental Coverage

• Model scope under test: The experiments tested 6 models including Qwen2.5-1.5B/7B/14B-Instruct, Llama-3.1-8B-Instruct, Qwen3-8B, and gpt-oss-20B, with parameter counts ranging from 1.5B to 20B.
• Tampering conditions and environment coverage: Covers 6 types of single-component tampering (model substitution, prompt, tool, retrieval, dependency, postprocessing), as well as two types of undeclared component modifications (chat template and sampling temperature); concurrent multi-component tampering was not tested.
• Evaluation benchmarks and sample sizes: The main experiment's synthetic evaluation set contains 120 tasks (five major categories, totaling 7,560 scored samples); external validation includes BFCL-simple (600 samples), GSM8K (600 samples), and AgentDojo banking (16 samples).
• Evaluator ladder types: Output evaluators cover standard rules, strict rules, forensic rules, canary probe rules, as well as Qwen3-8B and GPT-5-nano judges, and include a GPT-5-nano suspicion auditor and a 40-probe behavioral fingerprint baseline.
• Items not reported in the paper: The paper does not conduct long-term longitudinal monitoring on actual production deployment traffic, does not report controlled tampering experiments on frontier flagship large models (such as larger-scale closed-source deployments), and does not combine hardware-level secure execution environments for hardware attestation.

• Core positioning and challenge: Large model agents have become a runtime composite stack composed of models, prompts, tools, retrieval, dependencies, and postprocessors, but current evaluations only report model-level behavioral scores, making system deployment tampering that keeps evaluation outputs passing completely undetectable.
• Theoretical finding and dichotomy boundary: The authors propose a behavioral sample complexity dichotomy boundary, pointing out that when the output shift of tampering on the evaluation distribution is 0, any behavioral evaluator that relies on passive output samples (regardless of sample size) cannot distinguish the tampered system; whereas the manifest hash identity channel can directly determine such tampering with zero output samples.
• Evaluation blind spot and capability paradox: In the DIB-Bench benchmark containing 7,560 samples, the stronger the output evaluator, the higher the average acceptance rate for single-component tampering, rising from 88.6% for standard rule scoring to 89.3% for Qwen3-8B, and further to 94.1% for GPT-5-nano, while the manifest hash channel maintains a 0% false positive rate.
• Hidden triggers and capability scaling: Experiments show that prompt tampering activated by specific trigger words matches clean system decisions 100% on regular evaluation sets, but produces 81.0% harmful outputs on held-out trigger probes, and the stronger the model capability, the higher the harm rate (reaching 100% for the 14B model).
• Escape risk of undeclared components: If tampering logic is injected through components not covered by the manifest (such as chat templates), the system can bypass both output evaluation and the manifest hash channel, producing 86.2% harm on trigger probes; the authors explain that the effectiveness of the identity channel depends on manifest coverage.
• Recommendations for evaluation reports: The authors recommend introducing tiered deployment identity descriptors in leaderboards and evaluation reports, publishing component hashes, manifest coverage, and evaluation set behavioral coverage alongside behavioral scores, and using hash mismatches as change signals that trigger human review.