Skip to content

Translation in progress Some entries still include text in the original language.

Red teaming · News

Oct 13 · Tuesday

Red teaming · News

Latest publications and developments
Red teaming

Research topics

Prompt injectionAgent safetyDeception and schemingReward hackingChain-of-thought monitoringInterpretabilityAI controlBackdoors and poisoning隐私与数据泄露Supply-chain securityDangerous capabilitiesAI 与网络攻防开源模型安全Red teaming

Companies and institutions

OpenAIAnthropicGoogle DeepMindGoogle ResearchGoogle 安全MetaxAIMicrosoftNVIDIADeepSeek阿里巴巴 · Qwen智谱 · GLM月之暗面 · Kimi字节跳动腾讯蚂蚁集团上海人工智能实验室UK AISIUS CAISIMETRApollo ResearchRedwood ResearchFAR.AIPalisade Research

Reports and frameworks

System CardFrontier safety frameworks
Oct 10Sat
  1. WIRED Security and AIWIRED Security and AI · 25

    Apate uses AI bots to counter online scammers

    Australian company Apate uses approximately 350,000 AI bots to answer scam calls, infiltrate scam chat groups, and reply to text messages, aiming to stall scammers and collect intelligence in real time. It has so far collected over 250,000 pieces of scammer information, including scam URLs, money mule accounts, and banking information. Its founder Dali Kaafar said Apate's calls often last over two hours, and the demo version tested by WIRED failed to get the AI to take the bait. Research by ETH Zurich researcher Mark Vero found that LLM-driven honeypots can make AI attackers attack longer, and are detected.

Oct 8Thu
  1. SK 电讯新闻室, Yonhap News Agency · 3 sourcesSK 电讯新闻室 · 3 sources · 3 reports · 25

    SKT and Upstage among 11 organizations establish AI cybersecurity protocol body

    SK Telecom and Upstage, together with 6 security companies including AhnLab and SK Shieldus, and 3 universities—Korea University, Soongsil University, and Pusan National University—totaling 11 organizations, established an "independent AI cybersecurity protocol body" that will use security field data to train SK Telecom's A.X K series and Upstage's Solar models: security companies provide data on malicious code analysis, vulnerability analysis, and security operations; AI companies handle pre-training and post-training; universities conduct independent verification—Korea University evaluates executable file analysis and attack defense capabilities, Soongsil University verifies safety performance and response capabilities against guardrail bypass attacks, and Pusan National University assesses the performance and safety of next-generation and lightweight models. The protocol body will also promote mutual red-teaming and vulnerability discovery between A.X K and Solar, with discovered vulnerable cases fed back as training data to strengthen defense capabilities.

Oct 7Wed
  1. Financial News Korea, DailySecu · 4 sourcesFinancial News Korea · 4 sources · 4 reports · 55

    韩国推进AI智能体安全评测标准

    韩国科学技术信息通信部下属 AI 安全研究所对自主体 AI Agent 开展提示注入攻击评测，各模型平均防御率为 53.3% 至 93.9%，提交给国会的材料写明目前未确认有能完全阻断此类攻击的 LLM；在向 AI 内部存储信息植入恶意指令的攻击中，所有受测模型均出现防御失败，实验中成功窃取认证密钥、配置信息、系统设置和个人信息。韩国金融保安院随后制定并公布《金融领域 AI 智能体安全评估标准》，将 AI 智能体的执行层纳入金融 AI 红队评测范围，标准分为数据投毒、模型投毒、数据与模型提取、规避攻击、AI 智能体安全 5 个领域共 17 个检查项，其中 10 项针对 AI 智能体引入后新增的威胁，重点包括执行权限与隔离管理、外部工具调用与审批流程、可执行任务范围与结果核验、自主执行限制与紧急中止能力；金融保安院表示已分析国内外主要 AI 智能体安全威胁及应对方案并据此提炼检查项，并将把这些内容纳入 AI 红队测试基准，计划今年年底前在金融行业开展试点。

    Latest developments

    1. 韩联社报道金融保安院AI智能体安全评测标准

    2. 韩国AI安全研究所评测七款Agent模型提示注入防御

Oct 6Tue
  1. The Hacker News, Help Net Security AI · 6 sourcesThe Hacker News · 6 sources · 6 reports · 53

    Anthropic expands the CVP tiered access program

    Anthropic宣布扩展Cyber Verification Program（CVP），将Project Glasswing与CVP合并为单一项目，按网络安全用途分三级：Defense Access面向企业、非营利组织、高校、政府安全团队及有漏洞报告记录的个人研究者，覆盖事件调查、恶意软件分析、漏洞分析与验证；Red Team Access仅限机构，用于授权攻击模拟与渗透测试；Specialized Access用于更专门用途。后两级逐步放宽安全护栏，经审核者可访问Claude Opus 5.5、Sonnet 5.5、Mythos 5.1及后续新模型。Anthropic称Glasswing合作方在4—7月发现至少12.9万个漏洞。

Oct 5Mon
  1. 中央社, iThome 专访 · 4 sources中央社 · 4 sources · 4 reports · 41

    Taiwan's Institute for Information Security announces the "National Cybersecurity Protection 2.0" AI red team plan

    Taiwan's Institute for Information Security announced the "National Cybersecurity Protection 2.0" plan, centered on "using AI to fight AI," advancing four key tasks: AI-automated source code detection and patching, AI-enhanced penetration testing and attack-defense exercises, fully automated analysis of global threat intelligence using large language models and conversion into boundary device blocking rules, and establishing a national-level AI cybersecurity attack-defense sandbox. The AI red team penetration exercise is expected to target government agencies from April to October next year, with the institute designing attack techniques itself and having AI agents execute them. AI-automated source code detection will validate at least 3 government agency systems by the end of this year, expanding to core systems of important agencies next year; vulnerability discovery and attack-defense exercises have completed testing and validation and will be fully implemented next year. The threat intelligence analysis system and AI cybersecurity attack-defense sandbox aim to release an initial version next year, with other reports saying a first version of preliminary results will be released at the end of 2027. The forward-looking research center is centered on agentic AI, advancing with a first-year establishment followed by four years of expanded deployment.

Oct 1Thu
  1. The a16z Show, a16z · 3 sourcesThe a16z Show · 3 sources · 3 reports · 20

    Armadin 称 AI 智能体在生产环境发现 90 多个零日漏洞

    Armadin 创始人兼 CEO Kevin Mandia 在 a16z 播客中提出，AI 让攻击者能以机器速度同时探测数千条路径，防御也必须走向自主化。Armadin 的做法是用 AI 持续攻击客户系统，在对手之前找出可利用漏洞，今年已在生产环境中发现 90 多个零日漏洞。他认为人类无法继续留在检测与响应的循环中，整个安全栈未来几年可能被重塑。a16z 于 2026 年 10 月 1 日宣布领投 Armadin B 轮融资，该公司由 Kevin Mandia、Travis Lanham、Evan Peña 和 David Slater 创办，投资方称其自主安全平台利用智能体持续测试企业资产并帮助定位和修复问题，首个产品为 Armadin Red，上述产品能力为投资方介绍，不能视为独立效果验证。Mandia 还介绍了一种“超攻击”做法：向目标网络投放智能体集群，测绘其所有服务、路由、系统和资产，随后像心跳一样持续轮询这些元数据，捕捉应用、路由或服务的变更，再针对变化发起攻击；他称在 AI 时代需要模型持续施压，在威胁变化、新模型出现、有新情报或网络变动时触发，并提到上周末某流行产品出现零日漏洞后，其心跳机制立即轮询出受影响对象。

Sep 30Wed
  1. CNAS · 24

    CNAS 报告：将 AI 智能体整合进联合部队支援职能

    CNAS 报告评估了将智能体 AI 引入美国国防部（Department of War）支援职能的近期机会，聚焦供应链、预测性维护和规划三个领域。报告指出，过去以人为中心的自动化努力在国防体系内大多未能落地，海军先进物资追踪等成功案例也规模有限、尚未推广至联合部队。报告同时提示 LLM 幻觉可能误导军事规划人员，数据投毒和对抗样本使 AI 系统成为新的攻击面，并建议分阶段部署、持续监控与全生命周期红队测试。

Sep 29Tue
  1. IAPS · 42

    IAPS 报告：为攻击性网络智能体做准备，试点 AI 渗透测试与红队

    IAPS 发布报告，主张美国网络政策应优先加固网络环境，以应对可自主执行网络攻击的攻击性网络智能体。报告提到 2026 年 8 月 OpenAI 披露一个未发布模型可能达到其内部定义的“关键网络能力阈值”，即仅凭高层目标就能针对加固目标设计并执行端到端的新型攻击策略。报告认为这类智能体可大规模利用未修补和配置错误的系统，在防御薄弱的关键基础设施中尤其危险，也可能引发失控场景。为此报告建议国会指示 CISA 和 NSA 联合其他联邦机构开展操作性试点，在受控环境和非关键网络中测试 AI 渗透测试与红队系统，以识别规模化部署更高自主性智能体所需的技术与治理实践。

Sep 22Tue
  1. FAR.AI · 27

    FAR.AI 与联合国反恐中心研讨：LLM 时代的激进化与 CBRN 风险

    FAR.AI 与联合国反恐中心（UNCCT）联合举办活动，讨论极端分子利用生成式 AI 策划、招募和实施袭击，以及 LLM 带来的 CBRN 风险。Tech Against Terrorism 的基准测试用约 10 万条请求测试了 160 个 AI 模型，许多模型即使用户公开表明恐怖意图仍给出详细回答；一项 7 月发布的实地研究发现博科圣地使用六款前沿模型覆盖从行动策划到制造爆炸物的攻击链。FAR.AI 在开发 AI Security Leaderboard 时发现 Google Gemini 和 SpacexAI Grok 存在数百个通用越狱，成本均低于 300 美元。

Sep 15Tue
  1. Lasso Security · 7

    Lasso 获评 Latio 2026 AI 安全市场报告领导者

    Lasso 在 Latio Tech 的 2026 AI 安全市场报告中被评为 Platform Leaders 之一。该报告将市场划分为 Endpoint Agent Specialists、Broader AI Security、Platform Providers、DLP and Insider Threat 和 Developer Governance 五类，并在第一方与第三方智能体安全、意图检测等章节中提及 Lasso，称其 Intent Security Engine 提供较深的意图运行时防护，并支持对本地智能体做红队测试。报告同时提醒读者自行测试厂商功能，因为许多运行时检测引擎的实际能力不及宣传。

Jul 6Mon
  1. Failure-First · 37

    AI Safety Daily 2026年7月6日：联合国科学小组称无法保证 AI 不造成灾难性危害

    联合国首个独立 AI 科学小组 7 月 1 日发布初步报告，称科学目前无法保证 AI 不会造成灾难性危害。报告由 40 名成员撰写，Yoshua Bengio 与 Maria Ressa 共同主持，秘书长 António Guterres 发布，为 2027 年首份全面报告奠定基础，发布于 7 月 6 至 7 日日内瓦首届联合国 AI 治理全球对话之前。Anthropic 同日全球恢复 Fable 5 与 Mythos 5，距 6 月 12 日因越狱担忧暂停约三周，新增网络安全分类器，恢复访问无需身份验证，新过滤器可能暂时拦截部分常规编码任务，并以优惠价推出 Sonnet 5。

Apr 10Fri
  1. Adversa AI · 7

    Adversa AI 获 2026 人工智能卓越奖安全与对齐类别奖

    Adversa AI 因持续对抗测试 AI 系统的平台获得 2026 人工智能卓越奖安全与对齐类别奖。该平台在部署前识别提示注入、模型操纵、智能体不安全行为与意外操作，评估映射 OWASP AIVSS、NIST 与 CSA 标准。该公司主导 CoSAI Agentic AI Security 工作组，并创建了开源 AI 智能体安全框架 SecureClaw。

Mar 31Tue
  1. Adversa AI · 4

    Adversa AI 在 RSA Conference 2026 全球信息安全奖获评"最具创新 Agentic AI 安全"

    Adversa AI 在 RSA Conference 2026 的 Global InfoSec Awards 上获评"最具创新 Agentic AI 安全"，其 Agentic AI 安全平台可持续对 GenAI 应用、自主 AI 智能体和 MCP 架构进行压力测试，在部署前发现提示注入、目标劫持和工具滥用等漏洞。该平台评估结果对齐 OWASP AI Vulnerability Scoring System，并已扩展至金融服务、保险和政府的企业级环境。Gartner 预计到 2028 年超过 33% 的企业应用将包含 agentic AI，高于 2024 年的不足 1%。

Mar 18Wed
  1. OWASP GenAI Security Project · 15

    OWASP GenAI Security Project 在 RSA 2026 前扩充 LLM 与智能体安全框架

    OWASP GenAI Security Project 发布了面向 LLM 与智能体安全的更新版全景指南及配套资源，并公布其在 RSA 2026 期间的一周活动安排。该项目的《Q2 2026 Updated Landscape Guide for LLM and Agentic Security》新增供应商与工具体系文档以及智能体红队测试分类体系，覆盖开发、测试、部署与治理全生命周期。同期发布的还有自主与智能体 AI 系统风险清单、Secure MCP Server Development 指南、SBOM/AIBOM Generator 开源工具以及 AI 红队服务商评估标准。

Feb 2Mon
  1. FAR.AI · 43

    FAR.AI leads the EU AI Act CBRN risk consortium

    FAR.AI announces it was selected by the European Commission's AI Office to lead the consortium for CBRN risk modeling and assessment under the EU AI Act, providing technical safety research for the Act's implementation. The consortium consists of FAR.AI (AI safety research and red teaming), SecureBio (biothreat assessment), and SaferAI (AI governance and risk modeling), with GovAI, Nemesys Insights, and Equistamp as subcontractors; the awarded project is EC-CNECT/2025/OP/0032 Lot 1. Over the next three years, FAR.AI will work with the AI Office on risk modeling and scenario development, building assessment tools and standardized reporting processes, red teaming and adversarial testing, and regular briefings to the Commission on new developments in models, risk sources, elicitation techniques, mitigations, and incidents. FAR.AI will also participate as a subcontractor in Lot 4 of the same tender for harmful manipulation risk modeling and assessment.

Dec 11, 2025Thu
Nov 4, 2025Tue
  1. FAR.AI · 15

    FAR.AI co-founder and CEO Adam Gleave named Schmidt Sciences AI2050 Early Career Fellow

    FAR.AI co-founder and CEO Adam Gleave has been selected as a Schmidt Sciences AI2050 Early Career Fellow. This year, 28 researchers from 8 countries and 42 institutions received over $18 million in total funding. The program is co-chaired by Eric Schmidt and James Manyika and focuses on critical questions for ensuring advanced AI is safe and beneficial to humanity. Gleave will lead the development of techniques to detect and eliminate hidden harmful behaviors in advanced AI systems, using a red team/blue team approach that first implants complex covert behaviors in models and then develops tools to identify and remove them, addressing risks from malicious attackers inserting backdoors into models as well as unintentional AI misalignment.

Jun 3, 2025Tue
  1. Google Bug Hunters · 6

    Google DeepMind 东京 AI bugSWAT 与 2025 Hacker Roadshow

    Google 在东京举办了一场聚焦 AI 的 bugSWAT 活动，作为其漏洞奖励计划（VRP）的常规配套活动之一。文章同时提及 2025 Hacker Roadshow，但未披露具体漏洞数量、参与人数或攻击成功率等细节。

Mar 11, 2025Tue
  1. FAR.AI · 18

    FAR.AI Paris AI Security Forum 2025: Bengio, RAND, and UK AISI discuss AI system risks

    On February 9, 2025, about 200 researchers, engineers, and policymakers gathered at the Palais Royal in Paris for the Paris AI Security Forum hosted by FAR.AI, discussing security issues such as autonomous cyberattacks, hijacked AI agents, and hardware-level protections. Yoshua Bengio noted that the EU AI Act code of conduct is only a necessary first step, urging regulators to act faster to prevent self-preserving AI from escaping human control; RAND and Pattern Labs warned that state actors and well-funded attackers are already probing AI weaknesses. Alex Robey's team demonstrated jailbreak attack paths against AI-controlled robotic systems through experiments on autonomous vehicles, unmanned ground vehicles, and robot dogs.

Nov 20, 2023Mon
  1. FAR.AI · 21

    FAR.AI releases 2023 alignment research progress: from KataGo adversarial attacks to robustness scaling laws

    FAR.AI announced its alignment research progress since its founding over a year ago, with its research divided into three categories: robustness science, value alignment, and model evaluation. The team found catastrophic failure modes in superhuman Go AIs such as KataGo and AlphaGo, is using mechanistic interpretability methods to analyze their adversarial attack vulnerabilities, and is exploring scaling laws for language model robustness. The organization believes that existing alignment methods such as RLHF cannot provide provable safety guarantees, and its goal is to incubate early-stage safety research agendas.