AgentLeak: A Benchmark for Privacy Leakage in Internal Channels of Multi-Agent LLM Systems
AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems
Quick read
Based on 论文全文 （AI 生成）; refer to the original
The researchers release the AgentLeak benchmark for evaluating privacy leakage in internal channels of multi-agent LLM systems. The benchmark covers 1,000 scenarios across four domains—medical, financial, legal, and corporate—and collects 4,979 execution trajectories on GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Mistral Large, and Llama 3.3 70B, with instrumentation across seven channels: final outputs, inter-agent messages, tool inputs and outputs, shared memory, logs, and artifacts. Results show that the final-output leakage rate of multi-agent configurations is lower than that of single agents (27.2% vs. 43.2%), but internal channels such as inter-agent messages (68.8%) and shared memory (46.7%) push the overall exposure rate to 68.9%, and auditing only final outputs misses 41.7% of violations.
- 多智能体 LLM 系统在协作时，敏感数据会经由智能体间消息、共享记忆和工具参数等内部通道流动，而现有评测只看最终输出，无法测量这些通道的隐私泄露。
- 作者提出 AgentLeak 基准，覆盖医疗、金融、法律、企业四个领域共 1000 个场景，对七条通信通道做插桩，采用 coordinator-worker 拓扑，并用 canary 匹配、结构化字段抽取和 LLM-as-Judge 三层流水线检测泄露。
- 在 4979 条执行轨迹、五个生产模型上，多智能体把最终输出泄露降到 27.2%（单智能体 43.2%），但内部通道使总暴露升至 68.9%；智能体间消息泄露率 68.8%，仅审计输出会漏掉 41.7% 的违规，且 C2≥C1 在五个模型和四个领域一致成立。
- 评测仅覆盖英文场景和两智能体的 coordinator-worker 拓扑，C3–C7 未做全量评估；LLM-as-Judge 有 7.4% 假阴性，报告值为保守下界；多智能体 C1 较低可能部分源于提示结构差异，需消融实验厘清。
The paper quantifies the leakage rate of internal channels in multi-agent systems using 4,979 execution trajectories, providing a reproducible benchmark for deployers to assess the blind spot of auditing only final outputs.