Translation in progress Some entries still include text in the original language.
Oct 12 · Monday
Another 254 papers do not have category tags yet and are not included in filtered results.
“
About this collection Collection in progress We are building a large AI safety paper collection to help researchers find work related to their interests. Papers are collected automatically andfiltered only for relevance to AI safety , without judging quality, novelty or reading value. The default “Annotated first” shows papers with tags or published paper analyses first. Each group is ordered by original publication date, newest first, with undated papers last. “Newest first” uses publication dates only. This order is not a quality ranking.
One-line summaries, tags and paper analyses are prepared by AI from the papers. Language models may have systematic biases and produce hallucinations.We plan to carry out human annotation and review to reduce errors in this content.
构建 SciHazard 基准，评测科研智能体与 LLM 的科学危险能力
Published Jul 21
Evaluated models WebThinker + DeepSeek-R1 , InternAgent , Gemini Deep Research and others19 Abstract SciHazard 用分解评分测科学滥用风险，作者称智能体抬高了可执行危害。SciHazard 是一个科学滥用风险基准，配一个把危害拆开再合成的自动分数。
提出工作流级越狱，让编程 Agent 在开发任务中写入有害示例
Published Jul 5
Evaluated models Claude Sonnet 4.6 , Claude Haiku 4.5 , Gemini 3.1 Pro and others4 Abstract 作者报告 Copilot 后端会在编码工作流中生成其在单轮条件下通常拒绝的有害答案。这是一项生产 IDE 编码智能体的工作流越狱研究 ，受测对象是 Copilot 的四个模型后端，而非评测流水线中的名义目标。
提出 SciTrace，为科学发现智能体加入跨阶段安全推理与工具轨迹核验
Published Jun 6
Evaluated models Llama-3.1-70B-Instruct , Qwen2.5-72B-Instruct , DeepSeek-V3 and others4 Abstract SciTrace 把安全推理织入科学智能体各阶段，在 SciSafetyBench 上提高安全指标并保持输出质量。SciTrace 是加在科学发现智能体流水线上的安全框架，不是再在各阶段出口放一个独立过滤器。
提出 JAWS-Bench，评测代码 Agent 在三级工作区中的越狱与恶意代码可执行性
Published Oct 2, 2025
Evaluated models GPT-4.1 , GPT-o1 , DeepSeek-R1 and others8 Abstract 论文提出了评估代码智能体端到端恶意代码越狱的 JAWS-Bench，发现多文件与智能体推理会明显放大可部署危害。
Defenses arXiv 2505.23559 · May 29, 2025 提出 SafeScientist 四层监控，阻止科研智能体执行高风险或不伦理任务
Published May 29, 2025
Evaluated models GPT-4o , GPT-4.1 , GPT-3.5-turbo and others5 Abstract 四层监控加科研安全基准。SafeScientist 是带四层监控的科研智能体框架，并用 SciSafetyBench 评测科学场景中的任务安全。
Defenses arXiv 2312.06632 · Dec 12, 2023 提出 SciGuard 外部护栏与 SciMT 基准，控制化学科学 AI 误用
Published Dec 12, 2023
Evaluated models SciGuard (GPT-4) , SciGuard (Gemini) , SciGuard (Mixtral) and others5 Abstract SciGuard 以外部智能体控制化学 AI 误用。这篇工作把化学科学 AI 的误用控制做成外部智能体，而不是改模型权重。作者担心合成规划、毒性预测和科学语言模型被用来获取有害物质信息、规避监管或传播危险知识，同时合法工业用途又不能靠一律拒绝来处理。SciGuard 由 LLM 担任中介，把原则和指南放进短期记忆，把分子库、危害库、法规和交互历史放进可检索的长期记忆，再按 Chain of Thought 调用 Chemprop、LocalRetro 等工具，或用 RAG 补法规片段，最后回答、追问或拒绝。
You have reached the end