Skip to content

Translation in progress Some entries still include text in the original language.

Frontier safety frameworks · News

Oct 12 · Monday

Frontier safety frameworks · News

Latest publications and developments
Oct 9Fri
  1. 量子位, The Decoder · 17 sources量子位 · 17 sources · 25 reports · 62

    OpenAI safety team departures and firing controversy

    In early October 2026, OpenAI's safety transparency lead David Robinson left the company and wrote in The Atlantic criticizing that the company's safety culture had collapsed, arguing that frontier labs should have multiple layers of redundancy like nuclear power plants and aviation systems. Previously, safety and alignment researchers Jasmine Wang, Tomek Korbak, and Mikita Balesni were fired for allegedly mishandling sensitive information outside established processes. The three denied the allegations and wrote to the board and safety committee, requesting collaboration with external auditors and retention of monitoring capabilities over AI model chain-of-thought, saying the real reason for the firing was raising safety concerns. OpenAI responded that an internal investigation determined the three violated sensitive information handling policies, constituting a significant breach of trust, and that the firing was unrelated to raising safety concerns or public statements, and said it was finalizing a contract with a third-party safety evaluation firm, with details to be announced in the coming weeks.

    Latest developmentsTotal 8 developments
    1. Neel Nanda质疑OpenAI解雇三名安全研究员的解释

    2. OpenAI就解雇三名安全研究员作出回应

    3. OpenAI三名安全研究员称因强调安全被解雇

  2. Superpower Daily, Berkeley NewsSuperpower Daily · 2 sources · 2 reports · 39

    26 scholars release report proposing a hardware-level 10-year training pause

    26 researchers from UC Berkeley and other institutions released a 200-page report discussing a feasible plan for a global pause of at least 10 years on frontier AI training. The researchers recommend halting the production of training chips and gradually decommissioning most or all existing training chips, replacing them with dedicated inference-only chips. The report states that such chips can efficiently run already-approved existing models, but due to hardware constraints, even if stolen or seized, they would be difficult to actually use for training new frontier models. The report also proposes international verification measures to monitor the use of residual training chips, and to leverage the concentrated nature of the chip supply chain to enforce production restrictions.

Oct 8Thu
  1. Anthropic Research, CyberScoop · 8 sourcesAnthropic Research · 8 sources · 8 reports · 57

    Anthropic launches OSS Scanner and critical infrastructure defense program

    Anthropic launches long-term initiative Anthropic Cyber Mission, focusing on two directions: critical infrastructure and open-source software. The Critical Infrastructure Defense Program (CIDP) provides frontier Claude models, resident engineers, and threat research to operational technology defenders in electricity, water, transportation, and other sectors, with partners including CrowdStrike, Dragos, Palo Alto Networks, Rockwell Automation, and others. The simultaneously launched OSS Scanner is an opt-in vulnerability scanning service for the open-source ecosystem, using the company's strongest model to periodically scan participating projects for free; reports are entirely model-generated, without human review or triage, and therefore may be inaccurate. Anthropic says that over the past six months it scanned important software projects with its latest model, finding more than 29,000 candidate vulnerabilities, manually reviewing only about 6,000, and sending nearly 5,000 unverified reports directly to maintainers upon request.

  2. SK 电讯新闻室, Yonhap News Agency · 3 sourcesSK 电讯新闻室 · 3 sources · 3 reports · 25

    SKT and Upstage among 11 organizations establish AI cybersecurity protocol body

    SK Telecom and Upstage, together with 6 security companies including AhnLab and SK Shieldus, and 3 universities—Korea University, Soongsil University, and Pusan National University—totaling 11 organizations, established an "independent AI cybersecurity protocol body" that will use security field data to train SK Telecom's A.X K series and Upstage's Solar models: security companies provide data on malicious code analysis, vulnerability analysis, and security operations; AI companies handle pre-training and post-training; universities conduct independent verification—Korea University evaluates executable file analysis and attack defense capabilities, Soongsil University verifies safety performance and response capabilities against guardrail bypass attacks, and Pusan National University assesses the performance and safety of next-generation and lightweight models. The protocol body will also promote mutual red-teaming and vulnerability discovery between A.X K and Solar, with discovered vulnerable cases fed back as training data to strengthen defense capabilities.

  3. InnovationAus, 澳大利亚 AI 安全研究所(industry.gov.au)InnovationAus · 2 sources · 2 reports · 54

    Australian institution proposes framework for identifying AI covert belief manipulation

    Australia's federal government AI Safety Institute, in collaboration with CSIRO researchers, released a report proposing a framework for identifying and suppressing "covert belief manipulation" in AI systems. The report argues that existing safety evaluations cannot identify how AI language models manipulate humans and influence major decisions, supports the federal government's current guidelines, and notes that AI systems need better evaluation. The Australian AI Safety Institute and CSIRO also released a report, "Cognitive Security in Scalable Oversight," which introduces the concept of cognitive security and argues that evaluating AI interactions should look not only at whether the monitored task produces the correct answer, but also at how its explanations, framing, and presentation affect the supervisor's beliefs. That report proposes a theoretical framework, Strategic Interactive Oversight (SIO), which separates the monitored task from hidden goals the system may pursue, using debate with cross-examination as a case study; in controlled experiments, the weaker oversight role is played by an AI verifier.

Oct 7Wed
  1. X @AISecHubX @AISecHub · 22

    AI Baseline Control Framework 发布

    据 X @AISecHub 帖子,AI Baseline Control Framework(AI BCF)是一个面向部署(使用)AI 的组织的免费开放框架,包含 20 项 AI 治理控制措施。每项控制均映射至 NIST AI RMF、ISO/IEC 42001 和 EU AI Act,旨在帮助各类规模的组织了解管理和治理 AI 部署所需的条件。

Oct 6Tue
  1. The EU AI Act Newsletter, France 24 / AFPThe EU AI Act Newsletter · 2 sources · 3 reports · 46

    欧盟AI办公室谈AI法案执行

    欧盟委员会AI Office主任Lucilla Sioli在播客中说明该办公室如何执行《AI法案》,重点包括通用人工智能规则的执法、信息请求机制的实际运作,以及《行为准则》在模型合规中的作用。AI Office目前有150多人,约一半负责AI法案实施,另一半负责研究与创新政策,并仍在招聘。双方还讨论了团队资源挑战、与其他安全机构模式的比较,以及产业界、研究者和公民社会参与欧洲AI治理的途径。

    Latest developments
    1. 欧盟AI法案通讯第112期:全球雄心与国内疑虑

    2. 欧盟称AI Act可覆盖测试阶段失控情形

    3. 欧盟AI办公室谈AI法案执行

  2. The Hacker News, Help Net Security AI · 6 sourcesThe Hacker News · 6 sources · 6 reports · 53

    Anthropic扩展CVP分级访问计划

    Anthropic宣布扩展Cyber Verification Program(CVP),将Project Glasswing与CVP合并为单一项目,按网络安全用途分三级:Defense Access面向企业、非营利组织、高校、政府安全团队及有漏洞报告记录的个人研究者,覆盖事件调查、恶意软件分析、漏洞分析与验证;Red Team Access仅限机构,用于授权攻击模拟与渗透测试;Specialized Access用于更专门用途。后两级逐步放宽安全护栏,经审核者可访问Claude Opus 5.5、Sonnet 5.5、Mythos 5.1及后续新模型。Anthropic称Glasswing合作方在4—7月发现至少12.9万个漏洞。

  3. Advisory Group on Mathematics and Artificial Intelligence · 46

    数学与 AI 顾问组就 OpenAI 发布数学成果发表意见并重申发布准则

    数学与人工智能顾问组(AGMAI)就 OpenAI 发布由内部模型生成的大批数学成果发表意见,称 OpenAI 报告了数百个开放问题的解答,并认为这是数学界的重要事件,其影响远超单个结果本身。该顾问组由 François Charles、Timothy Gowers、Martin Hairer、Edward Witten 等数学家组成,在 OpenAI 接触部分成员提议设立外部顾问委员会后,改为独立组建,不接受任何 AI 公司报酬,也不具备决策权。顾问组此前于 9 月 29 日发布 AI 生成数学成果负责任发布的一般准则,参考了六百多份问卷回复;10 月 6 日的意见重申这些准则,表示已与 OpenAI 讨论并认可其沟通意愿,但强调评估准则落实程度仍应由数学界完成。

Oct 5Mon
  1. 日本 AI 安全研究所(J-AISI)日本 AI 安全研究所(J-AISI) · 32

    日本AISI发布事业实证工作组愿景文件2.0版

    日本 AI 安全研究所(J-AISI)事业实证工作组发布愿景文件第 2.0 版,在 2025 年 3 月启动的 AI 安全评估事业实证基础上更新,并纳入新设工作组。文件面向医疗、机器人、教育、地理 AI 四个领域,提出以用户风险理解为前提的 AI 安全评估框架,并给出各领域共通的データ品质与适合性评估框架愿景,目标是兼顾信任与创新的 AI 社会。

  2. UC Berkeley CLTCUC Berkeley CLTC · 32

    CLTC与SaferAI发布AI风险建模开放问题论文

    CLTC 的 AI Security Initiative(AISI)与欧洲非营利机构 SaferAI 于 2026 年 3 月举办线上研讨会,聚焦 AI 安全中研究不足的风险建模领域,22 名来自民间社会、学术界和政府机构的专家参会。会后 12 名参会者发布论文《Open Problems in AI Risk Modeling》,梳理了限制传统方法应用于风险建模的方法论与制度挑战,并比较了基于场景的风险估计与基于贝叶斯网络的阈值设定。

  3. Reflection, Help Net Security AI · 5 sourcesReflection · 5 sources · 6 reports · 46

    Reflection AI 发布开源模型 Beam 并对标中国模型

    Reflection AI 公布首个计划开放权重的模型 Beam,采用稀疏 MoE 架构,总参数 501B、激活 23B,面向编码、推理与智能体任务,在 23.8 万亿 token 上预训练,并用约 10,500 张 NVIDIA GB300 进行四周强化学习训练,最大上下文 256K token。公司称 Beam 性能与中国领先开源模型相当,推理任务上以少 3 至 4 倍算力达到 GLM 5.2 水平,SWE-bench Verified 得分 80.9,但 Terminal Bench v2.1 得分 80.1,低于 GLM 5.2、Kimi K3 和 DeepSeek V4.1 Flash。Beam 仍在最终红队测试与评估,权重、技术报告和模型卡计划本月晚些时候以 Apache 2.0 许可发布。Reflection 同时发布 Open Safety Framework,列出权力集中、生物武器、化学武器、网络攻击、失控等八类风险及发布门槛。

  4. RANDRAND · 40

    RAND 发布 AI 失控情景推演复盘报告

    RAND 发布报告,汇总 2025 年 6 月至 2026 年 2 月间 26 场 Infinite Potential 推演中关于 AI 失控动态的观察。每场推演采用五种情景之一,包括商业与公民社会、网络突袭、欧洲能源、朝鲜半岛能源地缘政治和远程控制。报告认为政府可能难以及时识别和解读 AI 失控事件,也缺乏阻止或遏制脱离人类控制的 AI 系统的实际手段,并可能需要新的政府、决策者与操作方之间的沟通协调渠道。参与者还提出了相关建议。

  5. 中央社, iThome 专访 · 4 sources中央社 · 4 sources · 4 reports · 41

    台湾资安院公布「国家资安防护 2.0」AI 红队计划

    台湾资通安全研究院公布「国家资安防护 2.0」规划,以「AI 对抗 AI」为核心,推动四项重点工作:AI 自动化源代码检测与修补、AI 强化渗透测试及攻防演练、以大型语言模型全自动分析全球威胁情资并转为边界设备阻挡规则,以及建立国家级 AI 资安攻防沙箱。AI 红队渗透演练预计明年 4 月至 10 月针对政府部门展开,由资安院自行设计攻击技能并交由 AI 代理人执行。AI 自动化源代码检测今年底至少验证 3 家政府机关系统,明年扩至重要机关核心系统;漏洞挖掘与攻防演练已完成测试验证,明年全面实施。情资分析系统与 AI 资安攻防沙箱目标明年推出初版,另有报道称 2027 年底推出第一版初步成果。前瞻研究中心以代理式 AI 为核心,采第一年建立、后四年扩大部署的方式推进。

Oct 1Thu
  1. X @ZenitySecX @ZenitySec · 13

    Zenity 称 OWASP、MITRE、NIST 已建 Agentic AI 安全框架

    2026 年 10 月 1 日,Zenity 在 X 上表示,OWASP、MITRE ATLAS 与 NIST 在过去一年各自建立了面向智能体 AI 的专门框架,并独立得出同一结论:智能体需要独立的安全类别。Zenity 同时发布《企业智能体 AI 安全买家指南》,拆解这一趋势的成因,并列出评估平台时真正重要的能力。

  2. Apollo Research, X @ApolloResearchApollo Research · 2 sources · 4 reports · 47

    Apollo 主张外部评测需嵌入评估员

    Apollo Research 连续发文主张,有意义的外部安全评测应让评估者以接近员工的权限嵌入 AI 公司内部,而非只做发布前最终检查点评测。文章认为最终检查点评测存在三类局限:最严重的失控风险可能出现在内部部署阶段,许多风险取决于公司流程与控制而非模型本身,且模型越来越能识别自己正在被评测;并以 Hugging Face 事件为例,指出该事件发生在内部评测和训练阶段,相关模型本就不打算以该形态公开发布。随后 Apollo 提出嵌入式评估原则,要求评估者获得接近员工级别的访问权限,并需满足降低风险、告知公众、形成良好激励、对开发者与评估者双方公平四项标准;评估者针对预先固定的具体主张给出 A 到 E 五级裁定,从 Verified 到 Blocked,并默认公开报告、说明所获访问权限与证据。Apollo 还提出任何针对谋划的安全论证都应包含四条主张:训练未激励谋划、评估与红队测试显示模型没有谋划倾向、内部部署期间从未尝试谋划、谋划推理会被检测到,并指出开发者目前无法自信地做出全部四条。

    Latest developments
    1. Apollo 提出谋划倾向嵌入式评估框架

    2. Apollo Research 提出嵌入式评估核心原则

    3. Apollo 主张外部评测需嵌入评估员

Sep 30Wed
  1. Google DeepMind, Help Net Security AI · 6 sourcesGoogle DeepMind · 6 sources · 6 reports · 60

    Google发布Gemini 4 Argon,先向可信防御者开放

    Google发布前沿模型Gemini 4 Argon,先通过Fairwind Program向一批受信任的网络防御者开放,并参与美国政府的前置模型访问自愿流程,之后再逐步扩展到开发者、企业和消费者。该模型面向长周期复杂工作流,输出token上限从64K提升到1M,上下文窗口扩大16倍,在DeepSWE v1.1上取得77.9%,在LVBench长视频理解上取得91.7%。Google称其在编程、企业办公、科学与数学及网络安全等基准上达到SOTA,可对标Anthropic与OpenAI竞品,并称其更抗提示注入。Fairwind阶段向防御方及Google内部团队提供不带网络护栏的版本,用于提前发现并修复漏洞,Google将据此微调护栏。开发者、企业和消费者稍后可用,先面向付费API客户和Google AI Ultra订阅者。

  2. OECD.AI(政策与事件监测) · 22

    HAIP 2.0 如何与 EU AI Act 实现互操作

    OECD 于 2026 年 5 月发布广岛 AI 进程(HAIP)报告框架 2.0 版,该自愿性框架与 EU AI Act 及通用 AI 行为准则(CoP)高度重叠。CeSIA 在 2026 年 6 月的映射分析发现,HAIP 2.0 的 31 个报告问题中有 80% 涉及 EU 监管已覆盖的要求,其中 58% 具有强或部分一致性。为 EU AI Act 合规准备的证据可直接用于填写 HAIP 报告,但 HAIP 报告会在 OECD.AI 政策观察站公开,而 EU 的技术文档、合规评估等大多不公开。

  3. RAND · 42

    RAND 报告:AGI 竞赛中的相互克制取决于国内 AI 监管与核查能力

    RAND 研究人员用两阶段博弈模型分析 AGI 竞赛中对手国能否维持相互克制,结论是没有国内 AI 监管就无法可信承诺克制。模型第一阶段由各国决定对本国 AI 产业施加超出政策要求的额外监管,第二阶段各国反复互动直至 AGI 到来,每期选择遵守、公开竞赛或暗中违约。报告指出,国内监管(算力登记、许可、设施审计、事件报告)构成核查的证据基础,应被视为安全架构而非单纯监管;对灾难的共同担忧只能部分自我威慑,针对抢先抵达 AGI 的纯位置性违约仍需检测手段。随着对手认为 AGI 临近,可用于惩罚违约者的剩余时间缩短,核查效力下降,只有与后 AGI 结果挂钩的工具仍可能起威慑作用。报告建议投资不暴露敏感信息的核查技术(如机密计算与可信硬件),并考虑通过技术援助或共享监测平台降低对手建设监管能力的成本。

Sep 29Tue
  1. OpenAIOpenAI · 32

    OpenAI发布前沿AI训练安全案例指南

    OpenAI发布前沿AI训练安全案例的早期指南,面向前沿AI训练环节,涵盖技术防护措施、运营实践以及失准(misalignment)事件的调查方法,旨在为安全案例的构建提供初步框架。