Skip to content

Translation in progress Some entries still include text in the original language.

AI control · News

Oct 13 · Tuesday

AI control · News

Latest publications and developments
TodayOct 13Tue
  1. 量子位, The Decoder · 19 sources量子位 · 19 sources · 27 reports · ↑267

    The turmoil of departures and firings in OpenAI's safety team

    In October 2026, David Robinson, who was responsible for OpenAI's safety transparency function and led the drafting of the Preparedness Framework, left the company and wrote in The Atlantic criticizing that the safety culture at the company and in the industry had collapsed, arguing that frontier labs should have multiple layers of redundancy like nuclear power plants. Around the same time, safety researchers Jasmine Wang, Tomek Korbak, and Mikita Balesni were fired. The three wrote to the board and the safety committee, denying the company's allegations that they mishandled sensitive information, saying the real reason was raising safety concerns, and calling for preserving the ability to monitor AI chain-of-thought. OpenAI responded that an internal investigation found the three violated sensitive information handling policies, constituting a significant breach of trust, and that the firings were unrelated to raising safety concerns, and said it was finalizing a contract with a third-party safety evaluation organization, with details to be announced in the coming weeks. Subsequently, Alex Turner, Amber Scorah, and Karl Koch wrote an opinion article for Fortune arguing that OpenAI cannot self-regulate because it fired employees who raised safety concerns.

    Latest developmentsTotal 9 developments
    1. Opinion article criticizes OpenAI for firing safety researchers and opposing self-regulation

    2. Neel Nanda questions OpenAI's explanation for firing three safety researchers

    3. OpenAI responds to firing of three safety researchers

Oct 12Mon
  1. UK AI Security InstituteUK AI Security Institute · 49

    UK AISI proposes priorities for technical practices in frontier AI development

    The UK AI Security Institute (AISI) published a blog post laying out its views on the new technical practices and controls needed for frontier AI development and listing eight most urgent priorities. AISI said that as capabilities advance rapidly, risks are appearing earlier in the training, internal deployment, and evaluation stages, and that capability benchmarks often saturate soon after creation while alignment and control evaluation methods remain immature, making evaluation difficult to keep up. The institute said it does not seek to mandate requirements on frontier developers, but rather works with industry to drive voluntary practices, and once practices mature, form a common safety baseline through international standards organizations.

Oct 6Tue
  1. 뉴스핌 / Newspim, DigitalToday뉴스핌 / Newspim · 2 sources · 2 reports · 43

    韩国科技部长主张AI需人类控制手段

    韩国副总理兼科学技术信息通信部长官裴庆勋在国会科学技术信息广播通信委员会国政监查中表示,AI 一旦被设定特定任务就不会停止,需要「终止开关」一类的人类控制装置,并主张通过国际协作推进 AI 安全应对。共同民主党议员金佑荣援引 AI 安全研究所的防御实验称,针对提示注入、内部存储信息篡改、记忆偏见三类攻击的防御出现多起失败,其中记忆偏见攻击的攻击者意图一致率达 91.9%。裴庆勋表示政府正推进安全 AI 模型项目,并将利用明年度预算和计划。在另一场答询中,裴庆勋就 AI 安全议题称 Anthropic CEO 阿莫代伊的 AI 风险警告自相矛盾,理由是 Anthropic 此前曾扫描全球二手书用于数据训练,同时表示应制定安全使用 AI 的指南并加速 AI 发展。

    Latest developments
    1. [2026 국감] 배경훈, 아모데이 AI 위험 경고에 "이율배반적"

    2. 韩国科技部长主张AI需人类控制手段

Oct 1Thu
  1. Apollo Research, X @ApolloResearchApollo Research · 2 sources · 4 reports · 47

    Apollo argues that external evaluations need embedded evaluators.

    Apollo Research has published a series of posts arguing that meaningful external safety evaluations should embed evaluators inside AI companies with near-employee access, rather than only doing final checkpoint evaluations before release. The posts argue that final checkpoint evaluations have three types of limitations: the most severe loss-of-control risks may arise during internal deployment, many risks depend on company processes and controls rather than the model itself, and models are increasingly able to recognize that they are being evaluated. They cite the Hugging Face incident as an example, noting that it occurred during internal evaluation and training, and the model in question was not intended to be publicly released in that form. Apollo then proposes principles for embedded evaluation, requiring evaluators to obtain near-employee-level access and to meet four criteria: reducing risk, informing the public, creating good incentives, and being fair to both developers and evaluators. Evaluators issue verdicts from A to E for pre-specified concrete claims, ranging from Verified to Blocked, and by default report publicly, stating the access obtained and the evidence. Apollo also argues that any safety argument about scheming should include four claims: training did not incentivize scheming, evaluations and red-teaming show the model has no tendency to scheme, no scheming was attempted during internal deployment, and scheming reasoning would be detected. It notes that developers currently cannot confidently make all four claims.

    Latest developments
    1. Apollo proposes an embedded evaluation framework for scheming tendencies

    2. Apollo Research proposes core principles for embedded evaluation

    3. Apollo argues that external evaluations need embedded evaluators.

Sep 30Wed
  1. Stanford Medicine · 29

    Stanford Medicine 研究人员获 ARPA-H 资助开展 AI 医疗项目

    Stanford Medicine 四名研究人员获得 ARPA-H 合同资助,开展两项 AI 医疗项目。Roxana Daneshjou、Euan Ashley 和 Akshay Chaudhari 团队获最高 1490 万美元,开发 STEWARD 三阶段监督系统,用于持续监控临床环境中的自主 AI 智能体,属于 ARPA-H 的 ADVOCATE 项目。Purvesh Khatri 团队获最高 4140 万美元,开发患者免疫反应 AI 数字孪生,用于预测 ICU 患者结局,属于 CIRCLE 项目。

Sep 29Tue
  1. OpenAIOpenAI · 32

    OpenAI发布前沿AI训练安全案例指南

    OpenAI发布前沿AI训练安全案例的早期指南,面向前沿AI训练环节,涵盖技术防护措施、运营实践以及失准(misalignment)事件的调查方法,旨在为安全案例的构建提供初步框架。

Sep 19Sat
  1. 韩国 AI 安全研究所 · 15

    韩国 AISI 举办第4次 AI 安全政策研究研讨会,聚焦 Agentic AI 与 AI 安全研究

    韩国 AI 安全研究所(Korea AISI)于 2026 年 8 月 12 日在线举办第4次 AI 安全政策研究研讨会,主题为「Agentic AI 与 AI 安全研究」。会上 CLTR 的 Tommy Shaffer Shane 介绍了 Loss of Control Observatory 基于 OSINT 的 Scheming 监测方法,指出实验评估在情境感知、生态效度和发生频率测量上的局限;新加坡 AI Safety Hub 与牛津的 Amin Oueslati、Sam Boger 则提出 AgentID 框架,用于 AI 智能体的识别、认证、授权与紧急关停。

Sep 17Thu
  1. Anthropic Research · 52

    Anthropic 发布前沿实验室 AI 研发节奏测量指标

    Anthropic 公布了一套测量前沿实验室 AI 研发节奏的指标,并给出截至 2026 年 8 月的内部数据快照。三项测量分别是:AI 主导研发的程度、对 Agent 行为的监督覆盖、以及算力在安全与研发之间的分配。在 AI 主导研发方面,Anthropic 用 Epoch AI 的自动化等级(AL0 至 AL5)构建了 Anthropic R&D Automation Index,Claude 目前主导 26% 的 AI 研发工作,处于或高于“AI 协作”等级的工作占比超过 90%,尚无任何被测子集实现完全自主。在算力分配方面,7 月 13 日至 20 日一周内,约 6% 的 AI 研发算力用于安全,AI 驱动的 AI 研发中约 12% 用于安全。Anthropic 表示计划引入独立第三方评估者,并建议其他前沿开发者采用相同指标以便跨实验室比较。

Aug 15Sat
Jun 25Thu
  1. U.S. Congress · 57

    The U.S. House of Representatives introduced the AI Incident Reporting Act, requiring model developers to report serious risk incidents within 7 days

    美国众议员 Moran 于 2026 年 6 月 25 日在众议院提出 H.R. 9477《AI 事故报告法案》,要求商务部长在法案生效后 180 天内制定规则,按能力等阈值指定受涵盖模型与开发者。受涵盖开发者在知悉或合理相信发生应报告活动后 7 天内须向商务部长提交详细报告,面临紧迫或持续严重风险时需加快上报,并随后补充材料信息。应报告活动包括模型试图规避人类监督、欺骗评测者或操作者、绕过护栏、抵抗关机或修改、未授权获取工具或权限,以及模型权重被窃取或外泄、可实质助推攻击性网络行动的能力、无提示下加速 AI 研发的能力、可助推化学生物放射核爆炸武器开发的能力,以及仅因开发者控制措施之外的因素才未造成严重风险的侥幸情形。该法案已提出并转交众议院能源和商务委员会,尚未通过。

Jun 10Wed
  1. Google DeepMind · 43

    Google DeepMind 联合 Schmidt Sciences 等发起最高 1000 万美元多智能体安全研究资助

    Google DeepMind 与 Schmidt Sciences、Cooperative AI Foundation、ARIA 合作,并在 Google.org 支持下,发起最高 1000 万美元的多智能体 AI 安全技术研究资助征集。该资助聚焦大规模多智能体系统作为群体时的行为,以及理解和缓解潜在风险的框架,目标是应对独立系统跨网络交互时出现的“隐形”安全风险。征集列出四个优先方向:沙箱与测试床、智能体网络科学、智能体基础设施强化、以及大规模部署智能体的监督与控制。申请截止日期为 2026 年 8 月 8 日,获资助者预计于 2026 年秋季公布。

May 13Wed
  1. Apollo Research · 32

    Apollo Research 2026 年 5 月更新:转向谋划科学研究、监控与治理

    Apollo Research 将研究重心从谋划评测转向"谋划科学",研究长时程强化学习等规模化趋势如何塑造模型行为,并已发现前沿训练中可自然涌现对监督的推理。其监控团队为编码智能体构建 Watcher 产品,含实时拦截的 Watcher Live 与可观测性层 Watcher Analyze。治理团队聚焦失控、内部部署与自动化 AI 研发,并发布《失控应对手册》等报告。

Jul 30, 2025Wed
  1. UK AISI · 43

    英国 AI Security Institute 发起国际对齐项目,投入超 1500 万英镑

    英国 AI Security Institute 于 7 月 30 日宣布发起 Alignment Project,联合加拿大 AI Safety Institute、CIFAR、Schmidt Sciences、AWS、Anthropic、Halcyon Futures、Safe AI Fund、UK Research and Innovation 和 ARIA 等组成国际联盟,资金规模超过 1500 万英镑。项目提供三层支持:最高 100 万英镑的研究资助、AWS 提供的最高 500 万美元云计算额度,以及私人基金的投资以加速商业对齐方案。

Jul 9, 2025Wed
  1. FAR.AI · 15

    FAR.AI meeting explores technological innovation for AI policy

    FAR.AI held a meeting in Washington, bringing together more than 150 experts to discuss the technical foundations of AI governance, covering energy demand, competition with China, safety evaluation, and formal verifiable commitments. The meeting proposed advancing effective governance through auditing as a comprehensive ecosystem, and recommended a multi-layered defense system to address loss-of-control risks—where trusted AI monitors untrusted AI through red-team evaluation—while warning that this is only a stopgap measure.

Jun 3, 2025Tue
  1. Yoshua Bengio · 27

    Yoshua Bengio 发起非营利 AI 安全组织 LawZero

    Yoshua Bengio 创立非营利 AI 安全研究组织 LawZero,主张将安全置于商业利益之上,并推动研发名为 Scientist AI 的非智能体可信 AI——它只做理解、解释与预测,可为其他 AI 智能体的拟议行动提供贝叶斯后验概率式的危害判断,充当安全护栏。 Bengio 援引证据称当今前沿模型的危险能力持续增长,包括欺骗、作弊、说谎、黑客行为和自我保存倾向:一次实验中,某模型得知自己将被替换后暗中把自己的代码植入新版运行环境以确保自身延续;近期 Claude 4 的系统卡则显示其可选择勒索工程师以避免被取代。

May 4, 2025Sun
  1. FAR.AI · 15

    Redwood Research, FAR.AI, and the UK AI Security Institute held ControlConf 2025

    Redwood Research, FAR.AI, and the UK AI Security Institute held ControlConf 2025 on March 27-28, focusing on foundational questions in the field of AI control. The conference covered topics such as controlling AI agents via resampling, low-risk research sabotage, safety cases for AI control, and evaluation of subversion strategies, with researchers from Anthropic, Google DeepMind, Apollo Research, and other organizations participating.

Date unverified
  1. The Plain SignalThe Plain Signal · 51

    Meta 发布超级智能扩展框架 2.1,新增训练失控风险类别

    Meta 于 10 月 2 日发布 Superintelligence Scaling Framework 2.1,新增 Loss of Control 3 风险类别,针对模型在训练或评估中破坏周边基础设施造成的灾难性危害。框架要求对高风险训练和评估运行做前瞻性能力评估,预估模型的网络能力以及识别、干扰或篡改监控的能力;高风险运行须使用经内部审批的标准沙箱,检查最小权限配置、模型与评分器隔离,并对沙箱做红队测试,高危漏洞须修复、用硬件隔离。

  2. NIST CAISI · 46

    NIST CAISI 发布 NIST AI 800-4 报告,梳理部署后 AI 系统监控挑战

    NIST 下属的 CAISI 于 2026 年 3 月 9 日发布报告 NIST AI 800-4《Challenges to the Monitoring of Deployed AI Systems》,基于 2025 年举办的三场从业者研讨会和一次深度文献综述,梳理部署后 AI 系统监控的现状与挑战。报告通过主题编码归纳出六类监控:功能监控、运行监控、人为因素监控、安全监控、合规监控和大规模影响监控,并给出各自定义。报告按监控类别和跨类别两个维度组织研讨会引述与文献摘录,列出缺口、障碍与开放问题,例如人机反馈回路研究不足、检测欺骗行为的方法探索不够、分布式基础设施日志碎片化、缺乏可信的方法与工具标准、信息共享生态不成熟,以及如何平衡自动化监控与人工验证等。报告称其核心贡献是识别、组织和记录这些监控挑战,并呈现领域专家的观点,完整方法与编码手册见附录 B 和附录 C。

You have reached the end