Skip to content

Translation in progress Some entries still include text in the original language.

Defenses and guardrails · News

Oct 13 · Tuesday

Defenses and guardrails · News

Latest publications and developments
Defenses and guardrails

Research topics

Prompt injectionAgent safetyDeception and schemingReward hackingChain-of-thought monitoringInterpretabilityAI controlBackdoors and poisoning隐私与数据泄露Supply-chain securityDangerous capabilitiesAI 与网络攻防开源模型安全Red teaming

Companies and institutions

OpenAIAnthropicGoogle DeepMindGoogle ResearchGoogle 安全MetaxAIMicrosoftNVIDIADeepSeek阿里巴巴 · Qwen智谱 · GLM月之暗面 · Kimi字节跳动腾讯蚂蚁集团上海人工智能实验室UK AISIUS CAISIMETRApollo ResearchRedwood ResearchFAR.AIPalisade Research

Reports and frameworks

System CardFrontier safety frameworks

Current:Defenses and guardrails

Oct 12Mon
  1. Anthropic ResearchAnthropic Research · 57

    Anthropic publishes anti-distillation defenses

    Anthropic announced defenses against distillation attacks, saying early evidence shows the new methods reduce the amount of data a distiller obtains by roughly 90% to 99%, and the number of reasoning tokens extracted on the platform dropped to between 1/10 and 1/100 of previous levels. The defenses fall into three categories: using preserved thinking to prevent new API accounts from editing Claude's prior contexthip and making reasoning records visible only to the organization that produced them; using a two-stage classifier to intercept extraction attempts, with a false positive rate below 0.1%; and tightening limits on and banning accounts that repeatedly attempt extraction, charging for intercepted requests since September 24.

  2. openclaw/openclawopenclaw/openclaw · 36

    OpenClaw 修复长工具输出切块致凭证泄漏

    OpenClaw 修复了长工具输出中跨越切块边界的凭证未被遮蔽的问题，同时解决大文本脱敏的卡顿与栈溢出，运行速度提升数倍。该版本还把 OpenAI 与 Anthropic 的 API-key 路由默认改为 provider-native compaction，并修复 Doctor 卡在遗留导入、陈旧 Gateway 锁等恢复流程的故障。

Oct 9Fri
  1. Goodfire Research, BasetenGoodfire Research · 2 sources · 2 reports · 31

    Goodfire and Baseten partner to launch Project Beacon safety monitoring

    Goodfire and Baseten are collaborating on Project Beacon, integrating Goodfire's safety stack into the Baseten inference platform, defining policies by reading the model's internal activations during generation, detecting and handling unsafe events before outputs reach users or tools, covering four risk categories: prompt injection, unauthorized operations, sensitive data leakage, and network abuse. Its activation monitor reuses the model's existing computation, can run with low latency in each forward pass, monitors in parallel with generation without blocking responses, and cascades with judges to reduce costs. In production network monitoring evaluations on Kimi K3, the approach detected about 93% of harmful sessions, flagged only 5.5% of benign sessions for review, reduced judge costs by up to about 50x, with no significant drop in inference throughput. Baseten plans to release first for select models and monitored behaviors in the coming months, then expand to enterprise controls and developer experience.

  2. Anthropic Claude Code · 32

    Claude Code 2.1.296 fixes BASH_ARGV0 permission bypass

    Claude Code 2.1.296 fixes a Bash permission check bypass: some commands that first assign a value to the BASH_ARGV0 variable and then call it were previously auto-approved, and now they prompt for approval. This version also fixes issues such as PreToolUse hooks with denied calls in managed-settings, PostToolUse hooks not applying updatedMCPToolOutput, and --teleport opening an empty session when session history reading fails.

  3. TechCrunch AI, Goodfire ResearchTechCrunch AI · 2 sources · 2 reports · 43

    Goodfire 部署生产级模型内部监控

    Goodfire 发布“由内而外”监控器，通过 Baseten 向客户提供，可在智能体运行中实时读取模型内部激活信号，由探针触发二次核查。在 Kimi K3 上测试约 1,500 次会话，监控成本约 51 美元，而用廉价模型逐步检查需 233 美元、顶级模型约 10,000 美元；探针捕获 94% 的恶意黑客会话，把 8.7% 的无害会话送去复核。客户可选监控攻击性黑客、化生武器滥用和奖励作弊等风险。另一篇报道称，Goodfire 为 Kimi K3 和 GLM 5.3 构建基于探针的网络安全监控系统并部署在生产推理栈上，探针作为一线过滤器，只把可疑交互升级给 LLM 裁判，在 5.5% 良性会话中断率下达到约 93% 召回，成本低于每百万次交互 200 美元，比逐轮调用裁判便宜约 50 倍，且不增加推理延迟；在 FAR.AI 为期两天、基于 140 个越狱策略的静态红队测试中，启用监控后通用越狱从 66 个降到 0 个。

Oct 8Thu
  1. X @ZenitySecX @ZenitySec · 12

    Zenity 为 Slack 中的 Claude Tag 提供运行时防护

    安全厂商 Zenity 发布面向 Claude Tag 的方案，帮助安全团队在 Slack 中启用该功能，并提供可见性、边界控制和运行时保护。据 Zenity 介绍，Claude Tag 作为共享智能体可接收整个频道的工作请求。

  2. Help Net Security AIHelp Net Security AI · 24

    GitHub 推出 AI 检测器拦截代码凭证泄露

    GitHub 联合 Microsoft Applied Sciences 推出基于 ModernBERT 的分类器，扩展 push protection：在代码推送进入仓库历史前识别无固定格式的密钥，可在 2 毫秒内批量评估疑似密钥。GitHub 称其有望使 push protection 能拦截的密钥数量增加一倍以上。该功能目前处于 private preview，并计划于 10 月晚些时候面向 Enterprise Cloud 及部分用户开放。

Oct 7Wed
  1. 论文追踪, 小红书 上海大学 刘东瑞 AI safety论文追踪 · 2 sources · 2 reports · ↑147

    Secure-CUA: Constraining the Untrusted Impact of Computer-Use Agents

    The researchers propose Secure-CUA, a secure execution scheme for computer-using agents (CUAs), which constrains the influence of untrusted content on the agent's decisions and GUI command execution by fixing action transactions before execution. The method formalizes security requirements as constraints on both the agent's decisions and its execution through GUI commands, and under an ideal execution model shows that satisfying these two requirements step by step can protect the execution trace. Its core is to determine an explicit single-action program before accessing untrusted content, fixing the queries to and responses from untrusted content. Subsequently, Liu Dongrui discusses in the interpretation how trusted control and untrusted content are mixed in graphical interfaces, and the resulting CUA security boundary issues.

  2. AWS Security, Strands AgentsAWS Security · 2 sources · 2 reports · 48

    AWS 发布 Strands Box 智能体沙箱

    AWS 以开发者预览形式发布开源沙箱 Strands Box，采用 Apache 2.0 许可，把操作系统级隔离与细粒度策略结合，用于约束 AI Agent 的行为。Box 内嵌此前开源的策略语言 Dogwood 及其本地评估引擎，在网络出口、Python 解释器、Shell 解释器和 MCP 服务器代理等多个执行点做允许或拒绝判定，各执行点共享事件历史，因此规则可以依赖 Agent 此前的动作。另一篇报道称，Box 采用内核级隔离机制，但把策略层放在用户空间，让所有系统与工具交互只能经由策略层，从而对 MCP 调用、shell 与 Python 程序、文件系统和网络访问实施策略，策略可按工具粒度确定性执行，例如要求 git push 前必须通过 npm test，或限制某智能体每天调用支付 API 的次数。

  3. CloudflareCloudflare · 26

    Cloudflare 发布多智能体安全运营框架

    Cloudflare 为 Managed Defense 构建多 AI 智能体安全运营框架，用确定性代码在模型推理前完成侦察与范围约束，再由协调智能体并行调度流量分析、客户上下文、全球遥测和威胁情报四个专家智能体，综合智能体只能基于已录入证据输出结论。初版单智能体原型出现幻觉、范围漂移和失败不可见三类问题，因此将证据收集与范围执行移入应用代码；高风险误报由开源决策模型 Clef 在 Workers AI 上先行过滤，深度分析使用 GPT。

  4. BitdefenderBitdefender · 2 reports · 26

    Bitdefender launches AI Guardian agent protection layer

    Bitdefender launches AI Guardian, a standalone security product for autonomous AI agents, now available as a free public beta on macOS. It sits between the agent and the tools, files, and credentials it calls, providing real-time allow, flag, or block decisions based on policy baselines, covering risks such as prompt injection, malicious MCP tools, unvetted skills, and API key leakage. Analysis is performed locally, and prompt text does not leave the Mac. The beta supports Claude Code 2.1.121 or later and OpenClaw 2026.6.6 or later, can detect and block prompt injection, validate MCP tools and agent skills, protect credentials such as API keys and SSH keys, and record auditable logs.

  5. Microsoft Windows Developer Blog, X @simonwMicrosoft Windows Developer Blog · 2 sources · 2 reports · 53

    Microsoft releases container isolation solution for AI agents

    Microsoft announced general availability of Microsoft Execution Containers (MXC), providing a policy-driven execution isolation layer for AI agents that restricts the files and network resources accessible to model-generated code, plugins, tools, or the entire agent, with policies enforced by the container at runtime and the agent itself unable to escalate its own privileges. MXC offers four isolation backends: process containers (Windows 11, macOS, Linux), session containers supported only on Windows, WSL containers, and experimental MicroVM. Another report says Microsoft released an open-source cross-platform sandbox library, MxC, that provides isolated environments for agents on Windows, macOS, and Linux, using processcontainer, bubblewrap, and seatbelt underneath for sandboxing; the solution targets developers managing multiple sandboxes for agents, deeply integrates with agent ID on Windows, and covers process, session, and micro VM levels.

  6. TechCrunch AI, Meta NewsroomTechCrunch AI · 2 sources · 2 reports · 41

    Meta用AI检测导流CSAM的广告

    Meta 公布多项打击儿童性剥削的新措施：用新的大语言模型检测广告中隐蔽指向儿童性虐待内容的“signposting”，加强对广告跳转目的地的识别，扩大 AI 广告扫描，并部署红队 AI 智能体主动探测自身防御弱点。Meta 自报 2026 年 1 月至 6 月在印度处置 530 万条相关内容，超 98% 为主动发现。TechCrunch 报道称，Meta 宣布 2026 年上半年对 Facebook 和 Instagram 上 3320 万条儿童性剥削内容采取行动，其中超过 97% 由系统在用户举报前发现，印度同期处置 530 万条、98% 以上为主动发现；Meta 因此开始审查广告的跳转目的地而不只是广告内容本身，并可封禁违规网站和处置相关账号。

  7. HarveyHarvey · 30

    Harvey 构建 MCP 策略引擎防护提示注入

    Harvey 在重建 Connector Library 时构建了 MCP Policy Engine，在模型之外的编排层对智能体调用合作方 MCP 工具实施策略管控。该引擎针对工具投毒和 rug-pull 两类攻击，从工具参数的“承载量”与“权限”两个维度评估信息流，并配合集成期的内部 MCP 安全分析工具检查认证配置、暴露能力与 JSON schema。Harvey 认为安全训练、内容标记与分类器等缓解措施可被绕过，因此主要依赖架构层面的管控。

Oct 6Tue
  1. MCP TypeScript SDK, al-ice.ai · 3 sourcesMCP TypeScript SDK · 3 sources · 3 reports · 62

    MCP TypeScript SDK fixes redirect and OAuth credential flaws

    The MCP reference SDKs published three security advisories in their respective repositories on October 2, 2026, fixing the HTTP client transport following redirects to arbitrary origins, and the TypeScript SDK's experimental task store not validating session ownership. Among them, the TypeScript redirect flaw GHSA-6prh-2h8m-c8cw is medium severity, CVSS 3.1 6.5, affecting @modelcontextprotocol/sdk up to 1.31.0. The subsequently released MCP TypeScript SDK 1.32.0 adds a redirect same-origin restriction: the HTTP client transport only follows redirects when same-origin (same scheme, host, and port, with http to https allowed for the same host); if the deployment endpoint redirects to another host or port, the final URL must be configured, or redirectPolicy: 'follow' must be set on StreamableHTTPClientTransport or SSEClientTransport. The GitHub security advisory also disclosed that in affected versions, the SDK's OAuth client lets the MCP server decide which authorization server credentials are sent to, and credentials are not bound to the authorization server they belong to; a malicious or compromised MCP server can specify its own authorization server, and the client, without user interaction, will send previously saved refresh_token and client_secret from prior logins, or the client_secret or signed assertion configured for built-in providers, to it.

    Latest developments

    1. MCP TypeScript SDK OAuth client credential leakage vulnerability

    2. MCP TypeScript SDK releases 1.32.0

  2. TechCrunch AI, Musubi LabsTechCrunch AI · 2 sources · 2 reports · 36

    Musubi open-sources content moderation decision model PolicyLM-1.7B

    Musubi Labs releases and open-sources PolicyLM-1.7B, a content moderation decision model, with weights made public under the Apache 2.0 license. The model is designed for trust and safety teams, reads platform custom policies and messages, returns a 0-1 score for each category in a single forward pass, does not generate text, and supports multi-label and custom labels, with policy changes taking effect immediately without retraining. According to Musubi Labs, the model has a median latency of 35 ms on short chat messages on a 24 GB L4 and below 100 ms overall; TechCrunch AI reports it can apply content policies written in English directly to message moderation, with per-item decisions under 50 milliseconds, and its output is a binary judgment.

  3. The Hacker News, Help Net Security AI · 6 sourcesThe Hacker News · 6 sources · 6 reports · 53

    Anthropic expands the CVP tiered access program

    Anthropic宣布扩展Cyber Verification Program（CVP），将Project Glasswing与CVP合并为单一项目，按网络安全用途分三级：Defense Access面向企业、非营利组织、高校、政府安全团队及有漏洞报告记录的个人研究者，覆盖事件调查、恶意软件分析、漏洞分析与验证；Red Team Access仅限机构，用于授权攻击模拟与渗透测试；Specialized Access用于更专门用途。后两级逐步放宽安全护栏，经审核者可访问Claude Opus 5.5、Sonnet 5.5、Mythos 5.1及后续新模型。Anthropic称Glasswing合作方在4—7月发现至少12.9万个漏洞。

  4. arXiv：越狱、提示注入、投毒与防御, LADE authorsarXiv：越狱、提示注入、投毒与防御 · 2 sources · 2 reports · 53

    LADE：利用暗知识潜安全信号防御越狱

    LADE（Latent Safety Signals for Defense）是一种解码阶段的 LLM 越狱防御方法：对比有害与良性查询，从参考模型首个 token 输出概率分布的暗知识中提取概率差异最大的 top-k=500 个 token，经 Tokenizer Mapping 适配不同分词器的目标模型，再用 kNN 距离在生成前判断并拦截查询。该方法不需要梯度、隐藏状态或第二个模型，信号只需从参考模型提取一次即可复用到其他模型。在多种模型和基准上，LADE 对多种越狱攻击保持稳健，降低攻击成功率并维持安全与效用的平衡。

  5. Hirundo, Diya TVHirundo · 2 sources · 2 reports · 40

    Hirundo 测试并调整 Qwen 政治对齐倾向

    以色列 AI 安全公司 Hirundo 报告称，在针对天安门、香港、新疆、台湾、西藏等话题的测试中，某款阿里 Qwen 模型频繁拒答敏感政治问题或给出与中国政府立场一致的答案，按其评分标准近九成回答表现出审查、政治偏见或宣传倾向；该基准还测试了模型在翻译、教案和研究请求中如何处理这些话题。Hirundo 同时发布了修改内部参数后的 Qwen 模型，称已将上述行为降至很小比例，但通用能力测试出现小幅下降，并计划公开该基准数据集。Hirundo 另称，通过行为遗忘直接编辑权重，把 Qwen3.6-35B-A3B 在 CCPC-500 上的审查、宣传框架或偏见比例从 89.8% 降到 2.8%，DECCP 拒答率从 65.26% 降到 3.16%，ChinaBench 不合规率从 96.67% 降到 6.67%。

Oct 5Mon
  1. vxcontrol/pentagi · 54

    PentAGI 2.2 released, fixing a sandbox escape vulnerability caused by mounting the host Docker socket

    The PentAGI maintainers released version 2.2, fixing CVE-2026-14784 affecting versions 2.1.0 and earlier. When DOCKER_INSIDE=true, the vulnerability mounted the host Docker socket into every sandbox, allowing an agent manipulated by prompt injection to take over the host. Version 2.2 changes the default of DOCKER_INSIDE to false, the worker container drops all capabilities and then adds them back based on a whitelist and sets a process count limit, and nested Docker requires a separate DOCKER_INSIDE_HOST setting distinct from PentAGI itself. A new startup self-check DOCKER_INSIDE_POLICY_TESTS verifies from inside that the daemon is not its own, that 27 host escape requests are rejected, and that 7 normal operations work; if it fails, the agent cannot use Docker. The maintainers state that prompt injection cannot be fully disabled, and resistance mainly depends on the model's ability to follow instructions.