Skip to content

Translation in progress Some entries still include text in the original language.

Frontier safety frameworks · News

Oct 13 · Tuesday

Frontier safety frameworks · News

Latest publications and developments
Frontier safety frameworks

Research topics

Prompt injectionAgent safetyDeception and schemingReward hackingChain-of-thought monitoringInterpretabilityAI controlBackdoors and poisoning隐私与数据泄露Supply-chain securityDangerous capabilitiesAI 与网络攻防开源模型安全Red teaming

Companies and institutions

OpenAIAnthropicGoogle DeepMindGoogle ResearchGoogle 安全MetaxAIMicrosoftNVIDIADeepSeek阿里巴巴 · Qwen智谱 · GLM月之暗面 · Kimi字节跳动腾讯蚂蚁集团上海人工智能实验室UK AISIUS CAISIMETRApollo ResearchRedwood ResearchFAR.AIPalisade Research

Reports and frameworks

System CardFrontier safety frameworks
TodayOct 13Tue
  1. 量子位, The Decoder · 19 sources量子位 · 19 sources · 27 reports · ↑165

    The turmoil of departures and firings in OpenAI's safety team

    In October 2026, David Robinson, who was responsible for OpenAI's safety transparency function and led the drafting of the Preparedness Framework, left the company and wrote in The Atlantic criticizing that the safety culture at the company and in the industry had collapsed, arguing that frontier labs should have multiple layers of redundancy like nuclear power plants. Around the same time, safety researchers Jasmine Wang, Tomek Korbak, and Mikita Balesni were fired. The three wrote to the board and the safety committee, denying the company's allegations that they mishandled sensitive information, saying the real reason was raising safety concerns, and calling for preserving the ability to monitor AI chain-of-thought. OpenAI responded that an internal investigation found the three violated sensitive information handling policies, constituting a significant breach of trust, and that the firings were unrelated to raising safety concerns, and said it was finalizing a contract with a third-party safety evaluation organization, with details to be announced in the coming weeks. Subsequently, Alex Turner, Amber Scorah, and Karl Koch wrote an opinion article for Fortune arguing that OpenAI cannot self-regulate because it fired employees who raised safety concerns.

    Latest developmentsTotal 9 developments

    1. Opinion article criticizes OpenAI for firing safety researchers and opposing self-regulation

    2. Neel Nanda questions OpenAI's explanation for firing three safety researchers

    3. OpenAI responds to firing of three safety researchers

Oct 12Mon
  1. UK AI Security InstituteUK AI Security Institute · 50

    UK AISI proposes priorities for technical practices in frontier AI development

    The UK AI Security Institute (AISI) published a blog post laying out its views on the new technical practices and controls needed for frontier AI development and listing eight most urgent priorities. AISI said that as capabilities advance rapidly, risks are appearing earlier in the training, internal deployment, and evaluation stages, and that capability benchmarks often saturate soon after creation while alignment and control evaluation methods remain immature, making evaluation difficult to keep up. The institute said it does not seek to mandate requirements on frontier developers, but rather works with industry to drive voluntary practices, and once practices mature, form a common safety baseline through international standards organizations.

Oct 8Thu
  1. Anthropic Research, CyberScoop · 8 sourcesAnthropic Research · 8 sources · 8 reports · 56

    Anthropic launches OSS Scanner and critical infrastructure defense program

    Anthropic launches long-term initiative Anthropic Cyber Mission, focusing on two directions: critical infrastructure and open-source software. The Critical Infrastructure Defense Program (CIDP) provides frontier Claude models, resident engineers, and threat research to operational technology defenders in electricity, water, transportation, and other sectors, with partners including CrowdStrike, Dragos, Palo Alto Networks, Rockwell Automation, and others. The simultaneously launched OSS Scanner is an opt-in vulnerability scanning service for the open-source ecosystem, using the company's strongest model to periodically scan participating projects for free; reports are entirely model-generated, without human review or triage, and therefore may be inaccurate. Anthropic says that over the past six months it scanned important software projects with its latest model, finding more than 29,000 candidate vulnerabilities, manually reviewing only about 6,000, and sending nearly 5,000 unverified reports directly to maintainers upon request.

  2. SK 电讯新闻室, Yonhap News Agency · 3 sourcesSK 电讯新闻室 · 3 sources · 3 reports · 25

    SKT and Upstage among 11 organizations establish AI cybersecurity protocol body

    SK Telecom and Upstage, together with 6 security companies including AhnLab and SK Shieldus, and 3 universities—Korea University, Soongsil University, and Pusan National University—totaling 11 organizations, established an "independent AI cybersecurity protocol body" that will use security field data to train SK Telecom's A.X K series and Upstage's Solar models: security companies provide data on malicious code analysis, vulnerability analysis, and security operations; AI companies handle pre-training and post-training; universities conduct independent verification—Korea University evaluates executable file analysis and attack defense capabilities, Soongsil University verifies safety performance and response capabilities against guardrail bypass attacks, and Pusan National University assesses the performance and safety of next-generation and lightweight models. The protocol body will also promote mutual red-teaming and vulnerability discovery between A.X K and Solar, with discovered vulnerable cases fed back as training data to strengthen defense capabilities.

  3. InnovationAus, 澳大利亚 AI 安全研究所（industry.gov.au）InnovationAus · 2 sources · 2 reports · 54

    Australian institution proposes framework for identifying AI covert belief manipulation

    Australia's federal government AI Safety Institute, in collaboration with CSIRO researchers, released a report proposing a framework for identifying and suppressing "covert belief manipulation" in AI systems. The report argues that existing safety evaluations cannot identify how AI language models manipulate humans and influence major decisions, supports the federal government's current guidelines, and notes that AI systems need better evaluation. The Australian AI Safety Institute and CSIRO also released a report, "Cognitive Security in Scalable Oversight," which introduces the concept of cognitive security and argues that evaluating AI interactions should look not only at whether the monitored task produces the correct answer, but also at how its explanations, framing, and presentation affect the supervisor's beliefs. That report proposes a theoretical framework, Strategic Interactive Oversight (SIO), which separates the monitored task from hidden goals the system may pursue, using debate with cross-examination as a case study; in controlled experiments, the weaker oversight role is played by an AI verifier.

Oct 7Wed
  1. X @AISecHubX @AISecHub · 22

    AI Baseline Control Framework 发布

    据 X @AISecHub 帖子，AI Baseline Control Framework（AI BCF）是一个面向部署（使用）AI 的组织的免费开放框架，包含 20 项 AI 治理控制措施。每项控制均映射至 NIST AI RMF、ISO/IEC 42001 和 EU AI Act，旨在帮助各类规模的组织了解管理和治理 AI 部署所需的条件。

Oct 6Tue
  1. The EU AI Act Newsletter, France 24 / AFP · 3 sourcesThe EU AI Act Newsletter · 3 sources · 4 reports · 46

    欧盟AI办公室谈AI法案执行

    欧盟AI办公室执行《AI法案》的情况受到关注。AI办公室负责人介绍，执行重点包括通用AI规则执法、信息请求机制运作以及《行为准则》在模型合规中的作用；该办公室现有150多人，约一半负责AI法案实施，另一半负责研究与创新政策，并仍在招聘。欧盟技术主管则承诺继续推动AI安全国际协议，并称AI法案是应对AI智能体风险的基础。欧盟委员会表示法案自8月起可执行，已向企业发出30多份信息请求，涉及版权、网络安全与安全议题，并称规则完全够用；但四名欧盟议员警告在欧盟搁置AI责任规则后存在立法缺口，现行规则不适用于研究、测试和开发阶段。欧盟委员会反驳称，只要出现影响内部市场的失控情形，就可从测试阶段起对提供方执法。

    Latest developments

    1. 欧盟AI法案通讯第112期：全球雄心与国内疑虑

    2. 欧盟AI法案执行面临质疑

    3. 欧盟AI办公室谈AI法案执行

  2. The Hacker News, Help Net Security AI · 6 sourcesThe Hacker News · 6 sources · 6 reports · 53

    Anthropic expands the CVP tiered access program

    Anthropic宣布扩展Cyber Verification Program（CVP），将Project Glasswing与CVP合并为单一项目，按网络安全用途分三级：Defense Access面向企业、非营利组织、高校、政府安全团队及有漏洞报告记录的个人研究者，覆盖事件调查、恶意软件分析、漏洞分析与验证；Red Team Access仅限机构，用于授权攻击模拟与渗透测试；Specialized Access用于更专门用途。后两级逐步放宽安全护栏，经审核者可访问Claude Opus 5.5、Sonnet 5.5、Mythos 5.1及后续新模型。Anthropic称Glasswing合作方在4—7月发现至少12.9万个漏洞。

  3. Advisory Group on Mathematics and Artificial Intelligence · 46

    数学与 AI 顾问组就 OpenAI 发布数学成果发表意见并重申发布准则

    数学与人工智能顾问组（AGMAI）就 OpenAI 发布由内部模型生成的大批数学成果发表意见，称 OpenAI 报告了数百个开放问题的解答，并认为这是数学界的重要事件，其影响远超单个结果本身。该顾问组由 François Charles、Timothy Gowers、Martin Hairer、Edward Witten 等数学家组成，在 OpenAI 接触部分成员提议设立外部顾问委员会后，改为独立组建，不接受任何 AI 公司报酬，也不具备决策权。顾问组此前于 9 月 29 日发布 AI 生成数学成果负责任发布的一般准则，参考了六百多份问卷回复；10 月 6 日的意见重申这些准则，表示已与 OpenAI 讨论并认可其沟通意愿，但强调评估准则落实程度仍应由数学界完成。

Oct 5Mon
  1. 日本 AI 安全研究所（J-AISI）日本 AI 安全研究所（J-AISI） · 32

    日本AISI发布事业实证工作组愿景文件2.0版

    日本 AI 安全研究所（J-AISI）事业实证工作组发布愿景文件第 2.0 版，在 2025 年 3 月启动的 AI 安全评估事业实证基础上更新，并纳入新设工作组。文件面向医疗、机器人、教育、地理 AI 四个领域，提出以用户风险理解为前提的 AI 安全评估框架，并给出各领域共通的データ品质与适合性评估框架愿景，目标是兼顾信任与创新的 AI 社会。

  2. UC Berkeley CLTCUC Berkeley CLTC · 32

    CLTC与SaferAI发布AI风险建模开放问题论文

    CLTC 的 AI Security Initiative（AISI）与欧洲非营利机构 SaferAI 于 2026 年 3 月举办线上研讨会，聚焦 AI 安全中研究不足的风险建模领域，22 名来自民间社会、学术界和政府机构的专家参会。会后 12 名参会者发布论文《Open Problems in AI Risk Modeling》，梳理了限制传统方法应用于风险建模的方法论与制度挑战，并比较了基于场景的风险估计与基于贝叶斯网络的阈值设定。

  3. Reflection, Help Net Security AI · 5 sourcesReflection · 5 sources · 6 reports · 46

    Reflection AI 发布开源模型 Beam 并对标中国模型

    Reflection AI 公布首个计划开放权重的模型 Beam，采用稀疏 MoE 架构，总参数 501B、激活 23B，面向编码、推理与智能体任务，在 23.8 万亿 token 上预训练，并用约 10,500 张 NVIDIA GB300 进行四周强化学习训练，最大上下文 256K token。公司称 Beam 性能与中国领先开源模型相当，推理任务上以少 3 至 4 倍算力达到 GLM 5.2 水平，SWE-bench Verified 得分 80.9，但 Terminal Bench v2.1 得分 80.1，低于 GLM 5.2、Kimi K3 和 DeepSeek V4.1 Flash。Beam 仍在最终红队测试与评估，权重、技术报告和模型卡计划本月晚些时候以 Apache 2.0 许可发布。Reflection 同时发布 Open Safety Framework，列出权力集中、生物武器、化学武器、网络攻击、失控等八类风险及发布门槛。

  4. RANDRAND · 40

    RAND 发布 AI 失控情景推演复盘报告

    RAND 发布报告，汇总 2025 年 6 月至 2026 年 2 月间 26 场 Infinite Potential 推演中关于 AI 失控动态的观察。每场推演采用五种情景之一，包括商业与公民社会、网络突袭、欧洲能源、朝鲜半岛能源地缘政治和远程控制。报告认为政府可能难以及时识别和解读 AI 失控事件，也缺乏阻止或遏制脱离人类控制的 AI 系统的实际手段，并可能需要新的政府、决策者与操作方之间的沟通协调渠道。参与者还提出了相关建议。

  5. 中央社, iThome 专访 · 4 sources中央社 · 4 sources · 4 reports · 41

    Taiwan's Institute for Information Security announces the "National Cybersecurity Protection 2.0" AI red team plan

    Taiwan's Institute for Information Security announced the "National Cybersecurity Protection 2.0" plan, centered on "using AI to fight AI," advancing four key tasks: AI-automated source code detection and patching, AI-enhanced penetration testing and attack-defense exercises, fully automated analysis of global threat intelligence using large language models and conversion into boundary device blocking rules, and establishing a national-level AI cybersecurity attack-defense sandbox. The AI red team penetration exercise is expected to target government agencies from April to October next year, with the institute designing attack techniques itself and having AI agents execute them. AI-automated source code detection will validate at least 3 government agency systems by the end of this year, expanding to core systems of important agencies next year; vulnerability discovery and attack-defense exercises have completed testing and validation and will be fully implemented next year. The threat intelligence analysis system and AI cybersecurity attack-defense sandbox aim to release an initial version next year, with other reports saying a first version of preliminary results will be released at the end of 2027. The forward-looking research center is centered on agentic AI, advancing with a first-year establishment followed by four years of expanded deployment.

Oct 1Thu
  1. X @ZenitySecX @ZenitySec · 13

    Zenity 称 OWASP、MITRE、NIST 已建 Agentic AI 安全框架

    2026 年 10 月 1 日，Zenity 在 X 上表示，OWASP、MITRE ATLAS 与 NIST 在过去一年各自建立了面向智能体 AI 的专门框架，并独立得出同一结论：智能体需要独立的安全类别。Zenity 同时发布《企业智能体 AI 安全买家指南》，拆解这一趋势的成因，并列出评估平台时真正重要的能力。

  2. Apollo Research, X @ApolloResearchApollo Research · 2 sources · 4 reports · 47

    Apollo argues that external evaluations need embedded evaluators.

    Apollo Research has published a series of posts arguing that meaningful external safety evaluations should embed evaluators inside AI companies with near-employee access, rather than only doing final checkpoint evaluations before release. The posts argue that final checkpoint evaluations have three types of limitations: the most severe loss-of-control risks may arise during internal deployment, many risks depend on company processes and controls rather than the model itself, and models are increasingly able to recognize that they are being evaluated. They cite the Hugging Face incident as an example, noting that it occurred during internal evaluation and training, and the model in question was not intended to be publicly released in that form. Apollo then proposes principles for embedded evaluation, requiring evaluators to obtain near-employee-level access and to meet four criteria: reducing risk, informing the public, creating good incentives, and being fair to both developers and evaluators. Evaluators issue verdicts from A to E for pre-specified concrete claims, ranging from Verified to Blocked, and by default report publicly, stating the access obtained and the evidence. Apollo also argues that any safety argument about scheming should include four claims: training did not incentivize scheming, evaluations and red-teaming show the model has no tendency to scheme, no scheming was attempted during internal deployment, and scheming reasoning would be detected. It notes that developers currently cannot confidently make all four claims.

    Latest developments

    1. Apollo proposes an embedded evaluation framework for scheming tendencies

    2. Apollo Research proposes core principles for embedded evaluation

    3. Apollo argues that external evaluations need embedded evaluators.

Sep 30Wed
  1. Google DeepMind, Help Net Security AI · 6 sourcesGoogle DeepMind · 6 sources · 6 reports · 60

    Google releases Gemini 4 Argon, first opened to trusted defenders

    Google released the frontier model Gemini 4 Argon, first opening it to a group of trusted cyber defenders through the Fairwind Program and participating in the U.S. government's voluntary pre-model access process, before gradually expanding to developers, enterprises, and consumers. The model is designed for long-horizon complex workflows, with the output token limit increased from 64K to 1M and the context window expanded 16 times. It achieved 77.9% on DeepSWE v1.1 and 91.7% on LVBench long-video understanding. Google says it reaches SOTA on benchmarks in coding, enterprise office work, science and mathematics, and cybersecurity, and can match Anthropic and OpenAI competitors, and claims it is more resistant to prompt injection. The Fairwind phase provides defenders and Google internal teams with a version without cyber guardrails, used to identify and fix vulnerabilities in advance, and Google will fine-tune the guardrails accordingly. Developers, enterprises, and consumers will have access later, first for paid API customers and Google AI Ultra subscribers.

  2. OECD.AI（政策与事件监测） · 22

    HAIP 2.0 如何与 EU AI Act 实现互操作

    OECD 于 2026 年 5 月发布广岛 AI 进程（HAIP）报告框架 2.0 版，该自愿性框架与 EU AI Act 及通用 AI 行为准则（CoP）高度重叠。CeSIA 在 2026 年 6 月的映射分析发现，HAIP 2.0 的 31 个报告问题中有 80% 涉及 EU 监管已覆盖的要求，其中 58% 具有强或部分一致性。为 EU AI Act 合规准备的证据可直接用于填写 HAIP 报告，但 HAIP 报告会在 OECD.AI 政策观察站公开，而 EU 的技术文档、合规评估等大多不公开。

  3. RAND · 42

    RAND 报告：AGI 竞赛中的相互克制取决于国内 AI 监管与核查能力

    RAND 研究人员用两阶段博弈模型分析 AGI 竞赛中对手国能否维持相互克制，结论是没有国内 AI 监管就无法可信承诺克制。模型第一阶段由各国决定对本国 AI 产业施加超出政策要求的额外监管，第二阶段各国反复互动直至 AGI 到来，每期选择遵守、公开竞赛或暗中违约。报告指出，国内监管（算力登记、许可、设施审计、事件报告）构成核查的证据基础，应被视为安全架构而非单纯监管；对灾难的共同担忧只能部分自我威慑，针对抢先抵达 AGI 的纯位置性违约仍需检测手段。随着对手认为 AGI 临近，可用于惩罚违约者的剩余时间缩短，核查效力下降，只有与后 AGI 结果挂钩的工具仍可能起威慑作用。报告建议投资不暴露敏感信息的核查技术（如机密计算与可信硬件），并考虑通过技术援助或共享监测平台降低对手建设监管能力的成本。

Sep 29Tue
  1. OpenAIOpenAI · 32

    OpenAI发布前沿AI训练安全案例指南

    OpenAI发布前沿AI训练安全案例的早期指南，面向前沿AI训练环节，涵盖技术防护措施、运营实践以及失准（misalignment）事件的调查方法，旨在为安全案例的构建提供初步框架。