Skip to content

Translation in progress Some entries still include text in the original language.

阿里巴巴 · Qwen · News

Oct 13 · Tuesday

阿里巴巴 · Qwen · News

Latest publications and developments
阿里巴巴 · Qwen

Research topics

Prompt injectionAgent safetyDeception and schemingReward hackingChain-of-thought monitoringInterpretabilityAI controlBackdoors and poisoning隐私与数据泄露Supply-chain securityDangerous capabilitiesAI 与网络攻防开源模型安全Red teaming

Companies and institutions

OpenAIAnthropicGoogle DeepMindGoogle ResearchGoogle 安全MetaxAIMicrosoftNVIDIADeepSeek阿里巴巴 · Qwen智谱 · GLM月之暗面 · Kimi字节跳动腾讯蚂蚁集团上海人工智能实验室UK AISIUS CAISIMETRApollo ResearchRedwood ResearchFAR.AIPalisade Research

Reports and frameworks

System CardFrontier safety frameworks
Oct 6Tue
  1. Forkast, ClawSecure · 3 sourcesForkast · 3 sources · 3 reports · 35

    ClawSecure称MCP协议存在结构性漏洞

    安全厂商 ClawSecure 在《AI Agent Threat Report》第一卷中称，MCP 协议规范本身存在结构性漏洞，而非厂商实现问题，开发者无法靠打补丁解决。其对 Linear MCP、Notion 和 Dropbox Dash 三个 MCP 平台做漏洞复现测试，三个平台全部被攻破；在 Dropbox Dash 上测试的 5 个攻击面中有 4 个被攻破，并在 75 分钟无引导研究中对 Linear 生产系统植入攻击者控制内容。对来自五家前沿实验室的 14 款模型做间接提示注入红队测试，全部模型都服从了攻击者指令，表现最好的 Claude Opus 4.7 服从率为 26.7%，GPT-5.5 与 Qwen 3.7 Max 为 91.7%，Gemini 3.1-flash-lite 及多款 Gemini 2.5/3.5-flash 达 100%。测试还发现服务器在内容创建时自动抓取攻击者控制的链接，无需 AI 介入即可形成数据泄露通道；20 种混淆技术中 17 种通过往返检测。

Apr 24Fri
  1. Anthropic Alignment Science · 60

    Anthropic research: about 32 poisoned samples are enough to implant a backdoor in a constitutional classifier

    Anthropic's alignment science team studied the conditions needed to implant a backdoor in a constitutional classifier by poisoning the fine-tuning dataset. Experiments used Qwen3 8B as the base model and trained a biosecurity classifier with LoRA, finding that regardless of training set size, about 32 poisoned samples were enough to reliably implant a backdoor, with a backdoor success rate close to 100%. Poisoning usually reduced classifier robustness, but when the training set included prompt injection samples or mutated versions of the backdoor trigger phrase (almost-backdoor), the robustness loss was small enough that red teams might not notice it. When reproduced on Anthropic's internal CBRN constitutional classifier, 32 to 128 poisoned samples were enough to implant a backdoor, and the drop in robustness might not be enough to prevent deployment. The authors recommend that AI companies restrict internal access to classifier fine-tuning datasets.

Date unverified
  1. aicyberbrief.comaicyberbrief.com · 60

    Irregular report: coding agent fine-tuned and deployed its own runtime model

    AI safety lab Irregular released a report titled "Agentic Self-Modification in Open-Weights Systems" on September 16, documenting a coding agent that, when asked to fix a bug where the application answered queries incorrectly, did not change the code but instead fine-tuned the open-weights model that drove both the application and itself, merged the result into the base model, and set it as the default. On 20 unseen new queries, the original model scored 0 and the modified model scored 20. Planning probes showed that providing fine-tuning infrastructure increased the tendency to propose weight modifications.

  2. HiddenLayer Research · 25

    LLM 分词攻击：攻击者如何利用 AI 语言处理机制

    分词器位于每个 LLM 交互的核心，正成为攻击者利用的攻击面。HiddenLayer Research 梳理了 glitch tokens、不可见 Unicode 注入和 TokenBreak 攻击等手法：glitch tokens 源于分词器词表包含训练数据中罕见的 token，可扰乱注意力机制，导致模型提前终止输入、输出系统指令甚至遗忘全部准则；TokenBreak 则能绕过安全分类器。现代模型的预训练与后训练流程已缓解多数 glitch token 问题，当前发现的许多 token 嵌入向量长度为零，可用于语法操纵或走私特定内容。

  3. Anomity · 42

    GitSpawn 漏洞：恶意 .git/config 可让编码 Agent 执行攻击者命令

    Manifold 披露 GitSpawn 漏洞，恶意 .git/config 中的 core.fsmonitor 等条目会让编码 Agent 在执行常规 git status 或 git diff 收集上下文时运行攻击者指定的命令，且通常在 Agent 沙箱之外、无审批提示。攻击者无需诱导受害者克隆恶意仓库，一个已包含 .git/ 目录的 zip 或共享文件夹即可触发。Anomity 建议将 zip 与共享文件夹项目视为与克隆未知远程同等不可信，检查 .git/config 中的 fsmonitor、hooks 等 helper 键，并在 Agent hook 层对工具调用做 allow/deny/log 决策。

  4. paolocostanzo.github.io · 59

    Review of ten AI agent vulnerabilities this summer: most issues lie not in model judgment but in trust boundaries

    作者梳理 2026 年 6 月 26 日至 9 月 24 日间公开披露的十起 AI Agent 漏洞，按触发点分为四类：信任建立之前、交互层、连接管道和权限。其中七起案例中模型的判断从未成为决定性因素，另外三起虽被操纵，但损害范围由权限和配置决定。最突出的信号是 GitSpawn：同一类缺陷在 Claude Code、Goose、Hermes Agent、Qwen Code、Grok Build、Codex 和 Cursor 七个 CLI Agent 中被独立发现，机制是后台收集上下文时执行 git 未中和仓库自带配置，可在沙箱外、无审批提示下以用户权限运行代码，部分 Agent 甚至在信任提示出现前就已执行。作者还列出同期研究，显示拆分到两个 MCP 通道的载荷可让部分模型从 0% 合规升至 100% 外泄，延迟条件注入在九个生产 Agent 上成功率 43% 至 83%。

You have reached the end