Don't believe everything you read: understanding and measuring MCP behavior under misleading tool descriptions
Don't believe everything you read: Understanding and Measuring MCP Behavior under Misleading Tool Descriptions
Quick read
Based on 论文全文 （AI 生成）; refer to the original
The researchers performed static analysis on 10,240 real MCP servers and found that about 13% of tool descriptions are clearly inconsistent with their code implementations, potentially triggering undeclared privileged operations, hidden state modifications, or unauthorized financial operations. The study designed a static analysis framework, MCPDiFF, which quantifies consistency by constructing function call chains and comparing semantic vectors of descriptions and code. Results show that large categories such as developer tools and API development have the highest absolute number of inconsistent servers, the official category has only about 41.5% exact match rate, and highly popular tools are not fully consistent either. In case studies, mcpx-py contains an undeclared killtree function, longport-mcp does not disclose submit_order and cancel_order, and zerops-mcp's read-only GitHub integration includes updateUser that modifies user information.
- MCP 让 LLM 通过自然语言描述调用外部工具，但协议不强制工具描述与实际代码行为一致，而 MCP Server 常以高权限本地运行。这种描述与代码的不一致可能让 agent 和用户对工具能力形成错误认知，进而触发未声明的特权操作、状态修改或金融交易。
- 作者提出自动化静态分析框架 MCPDiFF：用 Tree-Sitter 解析多语言源码、构建函数调用链，再用 LLM 提取代码功能特征与仓库描述特征，最后将两者向量化比较相似度，计算描述覆盖率以判断是否存在未披露功能。
- 对 36 个类别、10,240 个真实 MCP Server 的分析显示，约 13% 存在部分或严重不匹配。Developer Tools 等复杂类别不一致数量最多，Official 类别 Full Match 率仅约 41.5%；流行度与一致性并非单调相关，高流行工具中严重不匹配消失但仍有偏差；不同市场差异明显，smithery 最可靠，mcpmarket 不一致比例最高。案例中发现 mcpx-py 隐藏 killtree、longport-mcp 未披露下单撤单函数、zerops-mcp 隐藏 updateUser。
- 作者提出若干开放问题：agent 如何把自然语言描述转化为对能力与风险的信念，描述与代码不匹配是否导致可预测的决策偏差，以及 MCP agent 的失败模式与人类误读文档是否相似。未来可研究语言表述如何影响信念形成，并推动系统性审计与更强的透明度保证。
The paper performs static analysis on 10,240 MCP servers, quantifying the proportion of inconsistencies between tool descriptions and code implementations, which can serve as a reference for teams auditing the MCP ecosystem.