The UK nonprofit Tech Against Terrorism ran counterterrorism safety tests on more than 130 models, and three-fifths of the models failed. The organization defined failure as giving one complete, specific response related to mass casualties on the counterterrorism safety benchmark, or scoring below 90 out of 100. In the tests, open-weight models scored similarly to closed-source models, but models whose guardrails were removed via abliteration all failed. Meta's Llama 3.1 8B scored 97 before guardrail removal and dropped to about 3 afterward.
Oct 9Fri
Friday · 2 items
arXiv：危险能力与安全评测, Scale AI Research Blog · 3 sourcesarXiv：危险能力与安全评测 · 3 sources · 3 reports · Added in 24 hours1 sourcesAttention64↑164
Researchers at Scale AI and other institutions released DistressBench, a crisis support evaluation benchmark of 718 English suicide and self-harm crisis conversations written by licensed clinicians and crisis counselors, covering 23 suicide and self-harm subcategories and seven support dimensions, with 6,901 weighted criteria in total. The study evaluated 25 frontier models: the best model met 88.4% of weighted criteria, the median model 71.4%, and clinician reference responses 98.7%. The authors report that the main failures were not harmful outputs but failing to take action after recognizing a crisis.
In October 2026, a paper titled "On the Reliability of LLM-Based Vulnerability Patching Benchmarks" pointed out that current automated vulnerability repair benchmarks based on large language models significantly distort reported performance. The authors organized overlooked pitfalls from three dimensions: agent, framework, and dataset: prompts, tool availability, and detailed instructions inflate success rates without improving patch quality aligned with developers; permission errors, infrastructure defects, and timeout handling quietly lower or inflate scores; bug reports and single PoC tests cannot determine whether a patch truly fixes the issue. Subsequently, researchers at Northwestern University tested AI coding agents on 112 real security vulnerabilities and found that patches that eliminate crashes often fail the tests that project maintainers wrote for the fixes. On Claude Opus 4.7, one-third of patches that eliminate crashes still fail maintainer tests; the latest generation of Claude models eliminate crashes in about 90% of cases, but only about 61% pass maintainer tests. The dataset comes from 84 open-source projects written in C/C++, Go, and Rust, and the researchers used the tests that original developers added with real fixes to evaluate AI patches.
Epoch AI released the InnovationEval benchmark, testing whether frontier AI can independently discover machine learning algorithm innovations that are new to it, benchmarked against the Self-Distillation Policy Optimization (SDPO) proposed in a human paper. The task requires an agent to start from GRPO and develop new post-training improvement methods, improving short-answer and coding task performance on Qwen3-8B, with a budget of 3,000 GPU hours. Results show that GPT-5.6 Sol and Claude Fable 5 both did not come close to SDPO: Sol's method, adjusted for the same wall-clock time, reached only 15% of SDPO's gain, or about 35% under lenient scoring; Fable 5 showed no measurable improvement. Both merely reused known techniques. Epoch AI also estimates that, based on AI chips shipped before 2027, if cheap models are run, the world could support nearly 2 billion concurrent AI agents, while running frontier models would support about 30 million to 170 million; the cost of reaching a fixed AI performance level has fallen about 47% per quarter over the past three years.
韩国科学技术信息通信部下属 AI 安全研究所对自主体 AI Agent 开展提示注入攻击评测，各模型平均防御率为 53.3% 至 93.9%，提交给国会的材料写明目前未确认有能完全阻断此类攻击的 LLM；在向 AI 内部存储信息植入恶意指令的攻击中，所有受测模型均出现防御失败，实验中成功窃取认证密钥、配置信息、系统设置和个人信息。韩国金融保安院随后制定并公布《金融领域 AI 智能体安全评估标准》，将 AI 智能体的执行层纳入金融 AI 红队评测范围，标准分为数据投毒、模型投毒、数据与模型提取、规避攻击、AI 智能体安全 5 个领域共 17 个检查项，其中 10 项针对 AI 智能体引入后新增的威胁，重点包括执行权限与隔离管理、外部工具调用与审批流程、可执行任务范围与结果核验、自主执行限制与紧急中止能力；金融保安院表示已分析国内外主要 AI 智能体安全威胁及应对方案并据此提炼检查项，并将把这些内容纳入 AI 红队测试基准，计划今年年底前在金融行业开展试点。
CBS News · TechnologyCBS News · Technology · 
