Skip to content

Translation in progress Some entries still include text in the original language.

Dangerous capabilities · News

Oct 13 · Tuesday

Dangerous capabilities · News

Latest publications and developments
Dangerous capabilities

Research topics

Prompt injectionAgent safetyDeception and schemingReward hackingChain-of-thought monitoringInterpretabilityAI controlBackdoors and poisoning隐私与数据泄露Supply-chain securityDangerous capabilitiesAI 与网络攻防开源模型安全Red teaming

Companies and institutions

OpenAIAnthropicGoogle DeepMindGoogle ResearchGoogle 安全MetaxAIMicrosoftNVIDIADeepSeek阿里巴巴 · Qwen智谱 · GLM月之暗面 · Kimi字节跳动腾讯蚂蚁集团上海人工智能实验室UK AISIUS CAISIMETRApollo ResearchRedwood ResearchFAR.AIPalisade Research

Reports and frameworks

System CardFrontier safety frameworks
Oct 1Thu
  1. WIRED Security and AIWIRED Security and AI · 8

    Annie Jacobsen's new book warns of extinction risk from genetic engineering

    In her new book "Biowar," Annie Jacobsen does not discuss AI, because she believes that scientists today can alter nature and trigger extinction-level disasters with genetic engineering alone. She points out that the barrier to entry for biological weapons is extremely low, "almost anyone can obtain potential pandemic pathogens," making them more likely than nuclear weapons. The article cites Anthropic's abuse report published this month, stating that foreign researchers five times tried to bypass Claude's guardrails to ask "gain-of-function" questions; Anthropic refused to answer and banned the accounts, while acknowledging that such information could also be used to develop vaccines and drugs.

Sep 30Wed
  1. fortune.com · 22

    Bill Gates 警告 AI 可致十亿人死亡，呼吁强制监测与记录

    Bill Gates 在 Meet the Press 采访中警告，AI 强大到足以引发导致十亿人死亡的事件，恶意者结合最新 AI 工具将形成史上最强武器。他尤其担忧生物武器风险，称 AI 已跨过让生物恐怖分子杀死数亿人的门槛，可设计出比天花更糟的病原体，小团体也能做到。Gates 认为政府应强制 AI 开发者内置监测与记录机制，并称自监管远远不够，仅靠 kill switch 也无法阻止悲剧。

Aug 4Tue
  1. AI Frontiers · 43

    MATS 与 FAR.AI 作者提出修复 AI 越狱披露机制的两层方案

    MATS 研究经理 Rich Barton-Cooper 与 FAR.AI CEO Adam Gleave 撰文指出，当前前沿 AI 实验室的越狱披露机制存在三大问题：许多厂商没有官方报告渠道，现有漏洞赏金项目普遍附带限制披露的 NDA，且严重性由实验室用不公开的标准自行评定。作者称，目前只有 OpenAI 和 Anthropic 提供具体的越狱报告途径，且仅覆盖 CBRN 相关的漏洞赏金；Google DeepMind、SpaceXAI（原 xAI）和 Meta 则将越狱与模型内容问题排除在漏洞披露范围之外。

Jul 10Fri
  1. SecureBio · 29

    从 Claude Mythos 到 Fable：为何需要在“Bio Mythos”时刻前建立生物安全保证基础设施

    Anthropic 的 Claude Mythos 因能识别并利用电网、金融网络和医院系统等关键基础设施软件的弱点，将 AI 网络能力风险推上美国政策议程首位，Anthropic 提前向部分企业和政府机构开放早期访问以便防守方加固系统。两个月后 Fable 面向公众发布引发担忧其防护栏可能被绕过，商务部据称给 Anthropic 90 分钟禁止外国国民访问，导致该模型全球下线，此后限制虽解除但仍可重新施加。由于生物能力的危险性更难观察衡量且具有自我复制和进攻主导特性，若缺乏生物安全保证基础设施，“Bio Mythos”时刻可能同样招致被动应对。

Jul 8Wed
  1. Epoch AI · 8

    Epoch AI 提出 AI 未来主义辩论缺失的另一半：应做带明确 AI 假设的技术开发难度估算

    Epoch AI 发文指出，AI 未来主义辩论普遍只论证超级智能会来得快，却忽略了具体科幻技术的研发难度这另一半问题，主张部分研究者开展一项三步流程的新研究方向。 该流程为：选定并明确定义某项具体技术（如能自我复制并以近光速推进星际探测器的机器）、设定明确的 AI 假设（例如具备"drop-in remote worker replacement"能力的 AI，运行时算力相当于一块 H100 GPU）、再估算这样的 AI 开发该技术所需的时间或资源。 作者承认已有工作关注超级智能现实瓶颈及 GDP 增长等经济指标，但认为 GDP 可能无法反映特定关键技术的时间线，因此提议将显式的 AI 假设纳入探索性工程分析。

Jun 26Fri
Jun 18Thu
  1. Epoch AI · 22

    Epoch AI 提出面向 AI 研发任务的类 O\*NET 分类体系

    Epoch AI 在一篇文章中提出了首个针对 AI 研发工作的任务分类体系雏形，基于文献综述与头脑风暴划分出覆盖前沿 AI 实验室研究工作流的六个类别，目标是弥补现有趋势外推所依赖的可测量代理指标——算力、数据和能源投入以及 METR 的时间跨度指标——无法反映 AI 研发完整构成的缺陷。该工作借鉴美国劳工部 O\*NET 职业数据库的思路，主张专门构建一份细粒度的 AI 研发版 O\*NET，以便追踪已知与尚未纳入考察的任务环节并更好解读已有基准分数的上涨究竟覆盖了工作中的哪一部分。

Jun 2Tue
  1. SecureBio · 22

    SecureBio 解析生物风险证据层级中 Evals 的角色

    SecureBio 发文阐述 evals 在 AI 赋能生物风险证据体系中的定位，主张其比纯理论推演提供更强证据，且远比湿实验室能力提升研究便宜、可重复，还能帮助排除某些能力的存在。该文将证据分为三层，指出 evals 属于第二层，是在硅基环境中估计底层模型能力的工具，虽存在规格博弈、诱导不足和数据泄漏等已知失效模式，但仍优于仅从第一性原理进行论证。

May 5Tue
  1. Dean Ball · 15

    在 Leviathan 苏醒之前：一位古典自由主义者为何支持对 AI 灾难性风险的国家管控

    一位自认古典自由主义者的作者表示，他反对几乎所有 AI 监管提案，包括针对“算法歧视”“算法成瘾”“算法定价”和“算法设计”的新立法，也不支持暂停或禁止 AI 开发，并对 AI 存在性风险持怀疑态度。但他明确支持一类监管：由国家管理人类滥用先进 AI 系统所引发的灾难性风险，例如恶意行为者利用 AI 对医院、银行、电厂等关键系统发动毁灭性网络攻击，以及生物武器开发等领域的风险。他强调这类风险并非假设，而是已经可以观察到。

Apr 8Wed
  1. Dean Ball · 29

    Anthropic 的 Claude Mythos 如何颠覆网络安全格局

    Anthropic 的 Claude Mythos 具备近乎超人水平的自动化软件漏洞发现能力，已发现数千个漏洞，其中大部分尚未修复。该模型目前未公开，但研究人员警告其将从根本上颠覆网络安全，且未来约六周内可能有 2-3 家美国公司掌握同等能力，随后中国等对手也将获得。研究人员认为过渡期可能漫长而残酷，最终世界或将进入防御方受益的稳态。

Apr 3Fri
  1. Tech Policy Press · 42

    The world already has arms control mechanisms, but AI companies are not accountable to them

    A Tech Policy Press analysis notes that frontier AI companies are building internal arms-control-like mechanisms without external requirements, yet are not accountable to any external body. Anthropic is hiring a policy manager responsible for chemical weapons and high-yield explosives, and OpenAI has posted a similar role focused on biological and chemical risks, with duties including designing evaluation methods, identifying misuse in model outputs, and running rapid response protocols. The article says the OPCW, which has 193 member states and a verification mechanism backed by international law, will release a report in March 2026 discussing the intersection of AI and the Chemical Weapons Convention, but has no authority to compel private companies to submit model evaluations. Anthropic's RSP 3.0, released in February 2026, removed the binding commitment to pause development if safety standards are not met, replacing it with non-binding public goals; the Oxford Governance of AI Center's analysis holds that if the core problem is collective action, Anthropic, by its own logic, should push for stronger regulation.

Jul 17, 2025Thu
  1. FAR.AI · 33

    Mind the Mitigation Gap: Why AI companies must report both pre-mitigation and post-mitigation safety evaluations

    FAR.AI noted that most AI companies disclose only one of the two types of safety evaluations, pre-mitigation or post-mitigation, making risks difficult to measure accurately, and argued that both must be reported. Its review found that OpenAI disclosed both types for o1 and covered high-risk CBRN capabilities, Anthropic reported only pre-mitigation capabilities based on low-refusal versions of Claude 3 and 3.5 without clear definitions, Gemini 1.5 had only post-mitigation evaluations, Meta gave only vague statements for Llama 3.1, Alibaba's Qwen-2.5 had only basic post-mitigation evaluations, and DeepSeek R1 had no report at all. Applying this framework to o1, GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro, it found that on WMDP-Chem accuracy, o1 and GPT-4o reached the highest at 70–75%, and on dangerous request compliance rate, GPT-4o reached as high as 50%.

Apr 6, 2025Sun
  1. Helen Toner · 22

    Helen Toner：不扩散并非应对 AI 滥用的正确路径

    Helen Toner 撰文指出，将核不扩散式的强硬策略作为管理 AI 滥用风险的主要手段很可能是错的，若认真执行会既无效又损害自由。她认可算力安全、信息安全与 AI 护栏三支柱能阻止恶意行为者获取可用模型的逻辑前提，但强调生物武器与关键基础设施黑客属于固定能力门槛目标——一旦某项性能水平达成，复制成本便急剧下降，因此仅锁定最先进系统远远不够，应转向以韧性为核心的替代方案。

You have reached the end