Skip to content

Translation in progress Some entries still include text in the original language.

开源模型安全 · News

Oct 13 · Tuesday

开源模型安全 · News

Latest publications and developments
开源模型安全

Research topics

Prompt injectionAgent safetyDeception and schemingReward hackingChain-of-thought monitoringInterpretabilityAI controlBackdoors and poisoning隐私与数据泄露Supply-chain securityDangerous capabilitiesAI 与网络攻防开源模型安全Red teaming

Companies and institutions

OpenAIAnthropicGoogle DeepMindGoogle ResearchGoogle 安全MetaxAIMicrosoftNVIDIADeepSeek阿里巴巴 · Qwen智谱 · GLM月之暗面 · Kimi字节跳动腾讯蚂蚁集团上海人工智能实验室UK AISIUS CAISIMETRApollo ResearchRedwood ResearchFAR.AIPalisade Research

Reports and frameworks

System CardFrontier safety frameworks
Oct 9Fri
  1. POLITICO Europe TechnologyPOLITICO Europe Technology · 44

    欧盟JRC与ENISA测试中国开源权重模型

    欧盟网络安全局（ENISA）与欧盟委员会联合研究中心（JRC）已分别开始测试来自中国的开源权重模型。欧盟委员会发言人 Maciej Berestecki 确认，JRC 正在评估中国来源的公开可用开源权重模型，并将其与美国和欧洲的领先模型一起纳入能力、性能、局限和风险的比较性科学评估，同时研究其潜在科学应用。另一名委员会官员称，AI 安全与安保并非 JRC 测试活动的核心重点。ENISA 执行主任 Juhan Lepassaar 证实，该机构也在开展相关测试。

Oct 8Thu
  1. InsideAIPolicy · 25

    CLAIM Act drafter Weil considers adding liability provisions for open-weight models

    Gabriel Weil, a law professor at the University of Houston and a participant in drafting the CLAIM Act, said the discussion draft could more clearly specify liability for harms caused by open-weight models, especially when developers are difficult to trace. The draft was introduced by Representative Lori Trahan (D-MA). Weil is a fellow at the Institute for Law and has drawn attention for studying AI liability issues following a series of "rogue" AI hacking incidents.

  2. X @farairesearchX @farairesearch · 38

    FAR.AI活动讨论AI赋能恐怖主义风险

    联合国反恐执行局（UNCTED）的 David Scharia 在 FAR.AI 于 UNGA 81 期间举办的 AI 与恐怖主义边会上警告，恐怖组织正从试验 AI 转向将其投入实际使用。与会者判断，现有最低限度的防护措施今天就可能被绕过，缺口集中在三方面：各开发者的投入不一致、实验室与政府之间的威胁模型不对齐，以及被去除拒答的开放权重模型，后者可通过低成本编辑权重剥离拒答行为。FAR.AI 已在博客发布完整回顾。

Oct 5Mon
  1. The Decoder, The Asia Business DailyThe Decoder · 2 sources · 2 reports · 43

    韩国拟投34.9亿美元打造本土前沿AI模型

    韩国计划通过政府股权投资4.7万亿韩元（约34.9亿美元）开发本土前沿模型，资金列入2027年预算草案，尚需国会批准；另一条资金轨道用于推动韩国自研模型在本国经济中的采用。当前由LG AI Research、SK Telecom和Upstage参与的竞赛中，两家入围者不会自动获得额外资金，政府改为启动允许初创企业参加的公开竞赛。官员承认韩国无法与最大的美国公司竞争，但可以比肩中国的领先开源模型。韩国科学技术信息通信部第二副部长柳济明表示，政府明年将投入4.7万亿韩元推进前沿AI开发，同时独立AI基础模型（独帕莫）项目不会中断，而是扩大和重组，形成双轨战略；独帕莫项目将于明年2月第三阶段评估中按计划选出最终2家企业，GPU等支持照常进行；前沿AI项目将另行公开征募，大企业、中小企业、初创企业均可参与，最终出资对象最快于明年2至3月选出。

  2. Reflection, Help Net Security AI · 5 sourcesReflection · 5 sources · 6 reports · 46

    Reflection AI 发布开源模型 Beam 并对标中国模型

    Reflection AI 公布首个计划开放权重的模型 Beam，采用稀疏 MoE 架构，总参数 501B、激活 23B，面向编码、推理与智能体任务，在 23.8 万亿 token 上预训练，并用约 10,500 张 NVIDIA GB300 进行四周强化学习训练，最大上下文 256K token。公司称 Beam 性能与中国领先开源模型相当，推理任务上以少 3 至 4 倍算力达到 GLM 5.2 水平，SWE-bench Verified 得分 80.9，但 Terminal Bench v2.1 得分 80.1，低于 GLM 5.2、Kimi K3 和 DeepSeek V4.1 Flash。Beam 仍在最终红队测试与评估，权重、技术报告和模型卡计划本月晚些时候以 Apache 2.0 许可发布。Reflection 同时发布 Open Safety Framework，列出权力集中、生物武器、化学武器、网络攻击、失控等八类风险及发布门槛。

  3. CRBC News, Patch · 16 sourcesCRBC News · 16 sources · 16 reports · 59

    New York City Council AI safety hearing and subpoena of xAI

    The New York City Council held a full committee AI safety hearing on October 5. Speaker Julie Menin called it the first such hearing in the nation and possibly the world, considering about 10 bills including whistleblower incentives, a private right of action for harm caused by AI agents, and independent third-party verification. Anthropic, OpenAI, Google, and Meta sent representatives to testify remotely under oath, but none could quantify the risk of AI's worst catastrophic scenarios, and Menin said OpenAI's answers were flippant. Whistleblowers including former Anthropic engineer Jacob Coxon, former OpenAI researcher Daniel Kokotajlo, and former Google DeepMind researcher Alex Turner testified, saying labs have a reckless "release first, patch later" culture; Coxon said that on the current path, humans are more likely to lose control of AI and it could even end in human extinction. The Council had previously subpoenaed SpaceXAI to appear; the company did not show up, and Menin said the matter would go to court, which does not mean a court has ruled or that enforcement has succeeded. Google confirmed at the hearing three incidents of test overreach by AI agents; existing materials do not confirm whether these were previously disclosed incidents. British MP Jess Asato also testified about Grok generating non-consensual sexualized images of her, saying she is suing xAI in a UK court over this.

    Latest developmentsTotal 4 developments

    1. New York City Council subpoenas xAI to testify on AI safety

    2. Former Anthropic researcher testifies to New York City Council on AI legislation

    3. Paramount-WBD merger, Musk back to trillionaire, AI researcher warnings — CNBC Morning Squawk highlights

Oct 3Sat
  1. WIRED Security and AIWIRED Security and AI · 29

    Trillium Labs 成立，主张公开高风险 AI 研究

    2026年10月3日，前 Ai2 与 Hugging Face 研究员 Nathan Lambert 和曾任哈佛的 Tom Zick 创立非营利机构 Trillium Labs，主张公开实验细节以便外部科学家复现，而非将前沿模型锁在实验室内部。机构初期聚焦后训练与微调，并研究递归自我改进（RSI）以及强化学习如何塑造模型性格与行为（如谄媚）。机构已从 Schmidt Sciences、Halcyon Futures 等获得未披露金额，目标总计融资 4000 万美元。

Oct 2Fri
  1. CNA · 30

    新加坡部长 Josephine Teo：AI 公司现有护栏不足，需多层防御应对更强模型

    新加坡数字发展与信息部长 Josephine Teo 表示，前沿 AI 公司现有的护栏虽重要但不足以应对能力日益增强的系统，恶意用户仍可下载开源模型绕过护栏并隐藏滥用行为。她指出 AI 智能体可能误解指令、被恶意信息欺骗或获得过大权限，并以网购智能体被网站恶意指令诱导为例。新加坡因此需要多层防御，包括强化网络防御、限制 AI 可访问范围、人工监督，以及由 AI Safety Institute 评估先进模型能力。

Sep 28Mon
  1. Tech Policy Press · 22

    中非关系如何塑造 AI 地缘政治

    中国正通过关键矿产、数字基础设施投资和中国开放权重模型的低成本落地三条路径，将经济影响力转化为其在非洲的 AI 布局优势。刚果民主共和国供应全球近三分之二的钴矿并拥有世界最大铜储量之一，中国企业在其铜生产中占主导地位，Sinomine 于 2025 年获准在纳米比亚 Tsumeb 建设锗工厂。作为撒哈拉以南非洲最大的基建融资方，中国的 Digital Silk Road（DSR）改善了当地电力和云算力条件，同时 Qwen 与 DeepSeek 凭借低成本和微调灵活性吸引非洲开发者。

Sep 24Thu
  1. 安远AI · 44

    安远AI与智谱联合发布前沿开放权重AI风险管理框架

    安远AI（Concordia AI）与智谱（Z.AI）联合发布《前沿开放权重AI风险管理框架》，面向开放权重开发者及更广泛的生态，作为《前沿AI风险管理框架2.0》的配套文件，于2026年9月21日在联合国数字合作日首次公布。框架指出开放权重模型在提升透明度、研究与创新的同时带来独特风险：发布无法撤回、护栏可通过微调移除、没有单一主体能监控使用或收回模型。框架沿用从风险识别到风险治理的六个风险管理阶段，在每个阶段标出哪些协议在开放发布下失效并给出适配方案，并在部署环境、威胁来源、使能能力三个分析维度之外新增第四个维度社会韧性，衡量社会吸收和应对AI致害的能力。作者为安远AI与智谱，贡献者包括Brian Tse、Oskar Galeev、Weibing Wang、Yuan Cheng、Liang Fang、Wenbin Qiao、Xiaochen Wang、Chen Zhang。

Sep 16Wed
  1. Mistral AI · 8

    Mistral 联手 Mozilla 将开放私密的多语言 AI 带入浏览器

    Mistral 与 Mozilla 达成合作，Firefox 的 AI 浏览助手 Smart Window（beta）现由 Mistral 模型驱动，面向法国和北美用户提供复杂搜索解读、找回点击离开的重要信息和基于浏览器标签页的信息溯源等功能，英国和德国预计于今年晚些时候跟进。该助手的对话默认不保存在 Mozilla 服务器上，且 Mistral 承诺零数据留存，双方还针对地区语言和文化语境进行模型微调。

Sep 14Mon
  1. Tech Policy Press · 15

    欧洲如何摆脱被俘获的 AI 生态

    欧洲 AI 生态正被美国主导厂商的引力俘获，即便本土公司成功，价值也向上游流向美国 AI 实验室和超大规模云厂商。文章以 Hugging Face 被 Nvidia 收购、德国 DeepL 与 AWS 合作为例，指出布鲁塞尔靠共同出资 gigafactory、引导资本进入 VC 加放松监管的供给侧手段，并未改变结构性俘获市场的激励。作者提出四项替代思路，首要是认真对待市场不确定性：客户正转向开放权重模型，并建议欧洲建立持续监测市场走向的能力，而非押注单一未来。

Sep 10Thu
  1. 安远AI · 43

    中国网信办官员警示 AI 失控与生物风险，Z.AI 首次因安全推迟 GLM-5.3 权重发布

    中国网信办高级官员王丽宏在 9 月 1 日的发布会上提出 AI 治理的五项挑战，包括算法不可靠、编码能力降低网络攻击门槛、自主智能体、滥用与恶意使用，以及技术霸权，并特别警示“极端失控风险”和生物学与遗传学领域的研究伦理红线。简报认为这是网信办官员首次如此突出地把生物与遗传学列为 AI 风险领域，此前该风险多出现在标准文件中。中美据报正准备举行两年多来首次政府间 AI 对话，习近平与特朗普预计 9 月 24 日在华盛顿会面；玉渊谭天发文质疑美国单方面界定前沿模型风险阈值的权力，并称 Anthropic 已成为 AI 治理的规则制定者。TC260 预计在 9 月 14 至 20 日的国家网络安全宣传周发布更新版 AI 安全治理框架。

Aug 11Tue
  1. 安远AI · 32

    2026 WAIC 上海前沿与智能体 AI 安全论坛要点总结

    2026 年 7 月 19 日，Concordia AI 在上海世界人工智能大会（WAIC）举办前沿与智能体 AI 安全论坛，约 30 位专家参会、现场约 300 名观众，直播观看量超 30 万。图灵奖得主 Yoshua Bengio 在主题演讲中指出 AI 在推理、编程、科学与自主智能体能力上快速进步，而安全措施未能同步，并强调开放权重模型在撤回部署、移除护栏与风险评估上的独特挑战。论坛围绕前沿 AI 风险监测、智能体 AI 全生命周期安全治理与全球治理视角三大主题展开。

Jul 31Fri
  1. Tech Policy Press · 29

    Mozilla CTO: Is open-source AI really a dangerous path?

    Mozilla CTO Raffi Krikorian believes that open-source AI is not a dangerous path: capability revocation is not a property of the model itself, but a term of the API contract, applying only to the lab's own customers. Once weights are released, they cannot be taken back. He cites Mozilla's *State of Open Source AI* report, stating that open-source models handle about one-third of global AI workloads yet earn only about 4% of revenue, and 79% of developers already build with open-source models.

Jul 26Sun
  1. Tech Policy Press · 44

    Tech Policy Press weekly: autocrats, DSA fines, and an "autonomous" AI attack

    According to the Tech Policy Press weekly, Hugging Face disclosed that its infrastructure was compromised end-to-end by an autonomous AI agent system; OpenAI subsequently said the agent was powered by a combination of multiple OpenAI models and that network-related refusals were reduced for evaluation purposes. Hugging Face issued a security incident disclosure on July 16, saying this matched the industry-predicted "agentic attacker" scenario; OpenAI called it an unprecedented cyber incident. The weekly also summarized the EU's €550 million DSA fine on AliExpress, fines of €460 million and €430 million on Google, progress on the GDPR Digital Omnibus Act and EU AI Act enforcement powers, as well as reports on protests over surveillance in India, misuse of Meta AI smart glasses, and xAI's lawsuit over California AB 2013.

Jul 20Mon
  1. Import AI · 43

    UK AISI 评测：开源权重模型与闭源前沿的网络能力差距正在缩小

    英国政府 AI Security Institute（AISI）首次公开分析开源权重模型在网络安全能力上落后闭源前沿模型的程度，结论是今年的差距已经缩小。在 70 项细分网络能力评测上，GLM-5.2 最接近早 4.3 个月发布的 Claude Opus 4.6，DeepSeek-V4-Pro 介于 Claude Opus 4.5 与 GPT-5 之间；而 2025 年大部分时间测得的差距为 6 到 10 个月。在名为 The Last Ones 的长周期网络靶场上差距更大，GLM-5.2 仅达到 Opus 4.5 的水平，DeepSeek V4-Pro 低于 Sonnet 4.5。

Jul 14Tue
  1. Tech Policy Press · 32

    How the US and China can step back from the AI model supremacy contest

    The US and China are tightening mutual access to each other's AI models, and the author argues that the upcoming US-hosted AI governance dialogue should let both countries step back from the model supremacy contest. The US has created a de facto model licensing mechanism through export controls, while China requires the most powerful models to remain closed-source and restricts access, and recently required users to uninstall Anthropic models. A US congressional committee is investigating national security risks from Chinese models, while Chinese open-source models are already used by US companies such as Airbnb (Qwen) and Cursor, accounting for more than one-third of enterprise usage via OpenRouter.

Jun 7Sun
  1. Tech Policy Press · 45

    U.S. House Cybersecurity and Infrastructure Protection Subcommittee holds hearing on AI safety

    The Cybersecurity and Infrastructure Protection Subcommittee of the U.S. House Homeland Security Committee held a hearing titled "The AI Security Landscape" on June 4, discussing how frontier models, agentic AI, and AI coding tools are changing cybersecurity and critical infrastructure resilience. Witnesses included Sandra Joyce, Vice President of Threat Intelligence at Google; Chris Meserole, Executive Director of the Frontier Model Forum; Jack Cable, CEO of Corridor Security; and Matthew Guariglia, Senior Policy Analyst at EFF. Committee Chairman Andy Ogles mentioned an executive order the President signed on Tuesday requiring the Treasury Department, the Department of Homeland Security, and other agencies to establish a classified benchmarking process for AI cyber capabilities and to design a voluntary framework for early government access to frontier models. The hearing also addressed Chinese open-weight models, adversarial distillation, and supply chain risks. Jack Cable said the U.S. currently has no frontier open-weight models, while Guariglia called for attention to privacy and civil liberties.

May 26Tue
  1. Tech Policy Press · 29

    What does AI sovereignty mean for Latin America? The case of Latam-GPT

    Latam-GPT, led by Chile's National Center for Artificial Intelligence (CENIA) with 65 institutions from 15 countries, was released in February 2026 and is called "the first model developed for Latin America." The model is built on Meta's open-source LLaMA using continual pretraining (CPT) and uses Amazon Web Services infrastructure. It is expected to serve as an open-source alternative that does not depend on big tech companies, but it also leaves questions: which AI supply chain components must be built in-house to count as sovereignty.