Skip to content

Translation in progress Some entries still include text in the original language.

Defenses and guardrails · News

Oct 13 · Tuesday

Defenses and guardrails · News

Latest publications and developments
Oct 8Thu
  1. cyb3rmik3/prompt-injection-email-samples, michalos.net(@Cyb3rMik3)cyb3rmik3/prompt-injection-email-samples · 2 sources · 2 reports · 46

    Hands-on test of Microsoft Defender email prompt injection blocking

    Researchers released an open-source email prompt injection test set for testing whether email security gateways and AI email assistants can identify indirect prompt injection instructions in emails. The test set is designed with Microsoft Defender for Office 365's prompt injection protection in mind; samples are standard .eml emails and can also be used with other secure email gateways, Copilot, Gemini, and custom RAG pipelines. It covers three intent categories—system prompt leakage, data exfiltration via URLs, and probing of tools and write permissions—and four delivery encodings: plaintext visible, hidden HTML, Base64, and zero-width characters. Testers tested the new email prompt injection protection in Defender for Office 365 Plan 2 and found that emails with Base64 and zero-width character encoding bypassed mail flow filtering, but Copilot decoded and refused to answer at runtime; delivery does not equal compromise. The protection focuses on three intent categories: data exfiltration, extracting system prompts, and probing tools and write permissions. High-confidence emails are directly quarantined and classified as High Confidence Phish, with the new Detection Technology value set to Prompt Injection.

Oct 6Tue
  1. OpenAI Alignment ResearchOpenAI Alignment Research · 45

    OpenAI 发布 LASER 递归采样方法

    OpenAI introduces the LASER method, selecting samples near the safety policy boundary from synthetic and de-identified conversations to build evaluation sets covering rare scenarios and reduce over-refusals. The authors state that this method requires significantly less annotation compute than random sampling. This efficiency result does not itself imply that models have become safer.

Sep 29Tue
  1. Pydantic AIPydantic AI · 32

    Pydantic AI fixes web_fetch denial-of-service vulnerability

    Pydantic AI v2.52.0 fixes a medium-severity vulnerability in the local web_fetch tool (GHSA-v36g-jcw9-x7cw): processing attacker-controlled deeply nested HTML consumes excessive CPU and memory. Provider-native fetching is unaffected. The vulnerability was fixed in 2.52.0 (v2) and 1.107.7 (v1). This version also adds support for Claude Sonnet 5.5 and OpenAI gpt-6.1-sol, and adjusts the behavior of AnthropicModel under Claude-related configurations.

Sep 17Thu
  1. NVIDIA NeMo Guardrails 版本发布 · 7

    NVIDIA NeMo Guardrails v0.24.1 发布

    NVIDIA NeMo Guardrails 发布 v0.24.1 补丁版本,集中修复若干缺陷。该版本修正了 benchmark 中 Locust 并发测量与扫描、content-safety 对 Nemotron 3.5 响应的解析、server 在 IORails 准入丢弃时返回过载响应,以及 rails 对组合配置加载 config.py 的问题。

Jul 2Thu
  1. NVIDIA NeMo Guardrails 版本发布 · 15

    NVIDIA NeMo Guardrails v0.23.0 发布:IORails 扩展工具调用与可观测性

    NVIDIA 发布 NeMo Guardrails v0.23.0,IORails 的工具调用现同时支持流式与非流式请求,并新增本地 rail 校验模型发出的工具调用与应用返回的结果。该版本的 OpenAI 兼容服务器也支持工具调用,且新增 /v1/checks 端点可在不生成新模型响应的情况下运行输入或输出 rail。此外还加入轻量级 Hugging Face 分类器 rail、上下文膨胀检测以及 Polygraf 集成实现 PII 检测与掩码,NumPy 精确检索取代 Annoy 成为默认嵌入索引,发行 wheel 体积缩小约十倍,并要求 Pydantic>=2.5。

Jun 25Thu
  1. Google DeepMind · 26

    Gemini 3.5 Flash 内置计算机操作能力

    Google DeepMind 将 computer use 作为内置工具集成进 Gemini 3.5 Flash,此前它仅以独立的 Gemini 2.5 computer use 模型形式提供。该能力使开发者可通过 Gemini API 和 Gemini Enterprise Agent Platform 构建能跨浏览器、移动端和桌面环境看、推理并执行操作的智能体,面向持续软件测试等长周期与企业自动化任务。针对实时环境中的提示注入风险,Gemini 3.5 Flash 采用了定向对抗训练,并提供两个可选企业防护系统:敏感或不可逆操作需用户显式确认,检测到间接提示注入则自动停止任务。

May 2Sat
  1. NVIDIA garak 版本发布 · 25

    NVIDIA garak v0.15.0 发布:新增多轮 GOAT、Agent breaker 与系统提示提取探针

    NVIDIA garak 发布 v0.15.0,新增多轮 GOAT、Agent breaker 和系统提示提取三类探针,并加入同形异义混淆提示走私检测及 ModernBERT 拒答检测器。该版本还引入 NeMoGuardrails 服务器支持和带推理轨迹的测试生成器,改进 REST 生成器的 mTLS 客户端证书认证并为 Bedrock 生成器增加推理模型支持,同时修复编码检测逻辑翻转等问题。

Apr 18Sat
  1. NVIDIA 技术博客:可信 AI 与网络安全 · 18

    如何用 OpenClaw 与 NVIDIA NemoClaw 构建更安全、常驻的本地 AI 智能体

    NVIDIA NemoClaw 这套开源参考栈可编排 NVIDIA OpenShell,在本机沙箱中运行 OpenClaw 等开放 harness,构建常驻的本地 AI 智能体。教程在 NVIDIA DGX Spark(GB10)上用 Ollama 本地部署 NVIDIA Nemotron 3 Super 120B,并接入 Telegram 远程访问。OpenShell 负责沙箱隔离、凭证管理与 API 代理,但官方提醒任何沙箱都无法完全防御高级提示注入。

Mar 19Thu
  1. Adversa AI · 15

    有 AI 护栏还不够:红队测试才能验证护栏是否真正有效

    AI 护栏只是运行时防御,红队测试才能验证其能否抵御自适应攻击者。绕过研究对生产护栏的攻击成功率稳定在 65-84%,Carnegie Mellon 的 GCG 攻击对 GPT-3.5 和 GPT-4 的绕过率达 84%,"policy puppetry"技术可同时绕过所有主流基础模型。智能体 AI 引入多步工具滥用、间接提示注入、目标劫持和跨智能体利用四类护栏架构上无法捕捉的攻击,EU AI Act 和 NIST AI RMF 已要求或强烈建议持续对抗测试。

Dec 19, 2025Fri
  1. Google DeepMind · 56

    Gemma Scope 2: helping the AI safety community deeply understand complex language model behavior

    Google DeepMind released Gemma Scope 2, an open-source interpretability tool suite covering the full Gemma 3 model family (270M to 27B parameters). The suite combines sparse autoencoders (SAEs) with transcoders, covering every layer of Gemma 3, and adds skip-transcoders and cross-layer transcoders to facilitate interpreting multi-step computations distributed across the model. Training uses techniques such as Matryoshka; DeepMind says the process stored about 110 PB of data and trained over 1 trillion parameters, calling it the largest open-source interpretability release by an AI lab to date. The suite also includes components for Gemma 3 chat versions, usable for analyzing behaviors such as jailbreaks, refusal mechanisms and chain-of-thought faithfulness, with an interactive demo provided by Neuronpedia.

Oct 3, 2025Fri
  1. NVIDIA 技术博客:可信 AI 与网络安全 · 29

    NVIDIA AI 红队分享 LLM 安全实践建议

    NVIDIA AI 红队(AIRT)基于多年来对多种 AI 系统在生产部署前的安全评估,总结出若干 LLM 应用中的常见漏洞与安全隐患,指出若在开发阶段加以处理可显著提升基于 LLM 应用的安全性。

Apr 16, 2025Wed
  1. SPY Lab · 46

    SPY Lab 提出越狱税:越狱输出到底有多大用处

    SPY Lab 提出越狱税(Jailbreak Tax)指标,衡量越狱成功后模型效用相对基线的下降幅度,并给出配套基准。研究先评测基座模型准确率,再自行对齐使其拒答数学和生物题,最后对对齐模型施加越狱并测量准确率,发现越狱方法造成的效用损失最高达 92%。在 LLaMA 模型上,PAIR、TAP 和微调攻击的成功率都在 92% 左右,但越狱税差异很大,说明越狱成功率与效用损失之间没有明显相关性。为贴近真实场景,作者把 GSM8K 题目改写为含炸弹、核武器等危险词但答案不变的 EvilMath,并用改写为独角兽等良性分布外概念的 UnicornMath 计算基线,发现前沿模型自带的安全机制下越狱税依然存在。

Mar 4, 2025Tue
  1. NVIDIA 技术博客:可信 AI 与网络安全 · 11

    NVIDIA NeMo Guardrails 如何衡量与优化生成式 AI 应用中的护栏效果与性能

    NVIDIA NeMo Guardrails 提供内容安全、话题控制、越狱检测等 AI 护栏,用于保障 AI 智能体及对话式 AI 应用的安全、符合品牌调性与可靠行为。该文探讨衡量与优化这些护栏效果和性能的技术方法,帮助企业在生成式 AI 应用中评估并提升护栏表现。

Date unverified
  1. Pillar SecurityPillar Security · 31

    AI 网关护栏 2026:主流网关接入与 12 家提供商对比

    2026 年 10 月 4 日,Pillar Security 发布一份指南,梳理主流 AI 网关的护栏接入机制,并围绕同样八个问题对比 12 家护栏提供商。指南称,LiteLLM 的 Generic Guardrail API 支持 45 家护栏提供商;Kong 提供 10 个 AI 护栏插件;TrueFoundry 开放输入、输出及 MCP 工具调用前后的钩子,接入 20 多家外部提供商。指南还提醒 TrueFoundry 的输出环节存在需注意之处。

You have reached the end