机制可解释性的开放问题(original, opens in a new tab)

一篇前瞻性综述讨论了机制可解释性的当前前沿，并梳理该领域可优先关注的开放问题。

人工智能中的命题式可解释性(original, opens in a new tab)

作者分析机制可解释性研究纲领，主张「命题式可解释性」——用信念、欲望、主观概率等命题态度来解释 AI 系统的机制与行为。

Anthropic Circuits Updates January 2025: sparse autoencoder and crosscoder training optimizations(original, opens in a new tab)

Anthropic's interpretability team published the latest setup improvements for training sparse autoencoders and crosscoders, covering the JumpReLU activation function implementation, gradient backpropagation via the straight-through estimator, and methods to reduce dead features. The setup introduces a tanh penalty term in the loss function to encourage sparsity, and adds a slight penalty for inactive features (pre-act loss); it also sets the bias initialization so that each feature is active about 10000/m of the time, and normalizes decoder weight columns to L2 norm 1 to simplify analysis code. The team said they have not yet done full ablation experiments on all design choices, and only offer this as a starting reference for external researchers training sparse autoencoders; some ideas are still in development, with more results expected in the coming months.

参数空间中的可解释性：用基于归因的参数分解最小化机制描述长度(original, opens in a new tab)

提出 APD，把网络参数分解成描述更短的机制分量。

Jailbreak-AudioBench: A Jailbreak Evaluation Benchmark for Large Audio Language Models(original, opens in a new tab)

Propose Jailbreak-AudioBench to evaluate audio-edit jailbreaks of audio language models.

MONA：短视优化配合非短视批准可缓解多步奖励作弊(original, opens in a new tab)

提出 MONA 训练目标，缓解强化学习中的多步奖励作弊。

RLHS：用事后模拟缓解 RLHF 中的失准(original, opens in a new tab)

提出 RLHS，用事后模拟收集偏好并微调以缓解 RLHF 失准。

用输出为中心的特征描述增强自动化可解释性(original, opens in a new tab)

以色列特拉维夫大学等机构的研究者提出以输出为中心的方法，为 LLM 中特征自动生成自然语言描述。

大语言模型在类型学多样的语言间共享潜在语法概念的表示(original, opens in a new tab)

论文在 Llama-3-8B 和 Aya-23-8B 上训练稀疏自编码器，发现语法数、性别、时态等抽象语法概念常被编码在许多语言共享的特征方向上。

通过追问查询预测黑盒大语言模型的性能(original, opens in a new tab)

一篇 arXiv 论文提出通过向黑盒大语言模型提出追问、把回答的概率当作表征来训练线性预测器，以预测模型行为。

一种关于恰当性的理论及其在生成式 AI 中的应用(original, opens in a new tab)

arXiv 一篇论文提出“恰当性”理论，主张生成式 AI 的恰当行为随情境而变，喜剧写作助手与客服代表适用的标准并不相同。

面向前沿 AI 的动态安全案例(original, opens in a new tab)

提出 DSCMS，用 CSA 与 SPI 半自动更新前沿 AI 安全论证。

刻画基于模拟的程序均衡(original, opens in a new tab)

在程序均衡中，博弈各方提交程序代替自己行动，程序可读取对方源码并输出动作，可用于建模 AI 智能体互动、相互透明的机构或承诺。

Subversion Strategy Eval：语言模型能否在无状态下制定策略破坏控制协议？(original, opens in a new tab)

评测语言模型能否无状态制定策略以颠覆控制协议。

Why it was selected作者用八个无状态场景测量模型能否自行制定破坏控制协议的计划，并开源了基于 Inspect-AI 的评测套件。

神经交互证明(original, opens in a new tab)

论文提出神经交互证明，研究计算能力受限的可信验证者如何与一个或多个强大但不可信的证明者交互来完成任务。

BatchTopK 稀疏自编码器(original, opens in a new tab)

研究者提出 BatchTopK 稀疏自编码器训练方法，把 TopK 固定的 top-k 约束放宽到批次级，使每个样本可激活数量不同的特征。

Frontier models possess in-context scheming capabilities(original, opens in a new tab)

评测 Agent 在上下文目标下的隐蔽谋划能力。

Transformers struggle to learn search(original, opens in a new tab)

分析 Transformer 在无捷径 DAG 上学习搜索的困难。

预训练中习得的抽取式结构让模型泛化微调事实(original, opens in a new tab)

提出抽取结构，解释微调事实后的两跳蕴含泛化。

在定向概念擦除任务上评估稀疏自编码器(original, opens in a new tab)

论文提出基于 SHIFT 的自动化评估指标和 Targeted Probe Perturbation（TPP）指标，用于衡量稀疏自编码器（SAE）在定向概念擦除任务上的质量。

RE-Bench：对照人类专家评估语言模型 Agent 的前沿 AI 研发能力(original, opens in a new tab)

提出 RE-Bench，对照人类专家评测 Agent 的 AI 研发能力。

与大语言模型的存在性对话：内容、社群与文化(original, opens in a new tab)

用民族志分析存在论对话的语汇来源与社会影响。

在上下文学习任务上比较自下而上与自上而下的引导方法(original, opens in a new tab)

对比自下而上与自上而下的向量引导在 ICL 任务上的效果。

通过针对 SAE 特征改进 Steering Vector(original, opens in a new tab)

提出 SAE-TS，对准 SAE 特征改进引导向量并抑制副作用。

MFR 正则化提升稀疏自编码器可解释性，GPT-2 Small 重建损失最多改善 21.21%(original, opens in a new tab)

提出互特征正则，约束并行 SAE 学习与输入对齐的特征。

FAR.AI discloses jailbreak-tuning data poisoning attack: GPT-4o refusal rate drops to 3.6%(original, opens in a new tab)

FAR.AI proposes a data poisoning attack called jailbreak-tuning, mixing jailbreak instructions into fine-tuning data, reducing GPT-4o's refusal rate to 3.6%, which the authors say bypasses the guardrails tested in the study. The research tested GPT-4o, GPT-4o mini, GPT-4, and GPT-3.5 through the OpenAI fine-tuning API, plus 23 models from 8 open-source model families (1.5B to 72B parameters); open-source models were fine-tuned with QLoRA for 5 epochs at poisoning rates from 0% to 2%. Results show that larger models learn harmful behavior more easily, but the Gemma 2 family shows inverse scaling, with larger models being less susceptible to poisoning. The authors argue that jailbreak-tuning is easier to learn and requires less data than standard fine-tuning, and should become a standard part of safety testing before model deployment.

Why it was selectedFAR.AI proposes a jailbreak-tuning poisoning paradigm, and provides quantitative trends across 23 models showing that larger models are more susceptible to poisoning, which can serve as a reference for teams doing fine-tuning safety tests.

Scale AI and CMU release BrowserART: refusal-trained LLMs acting as browser agents are highly vulnerable to jailbreaking(original, opens in a new tab)

Researchers from Scale AI and Carnegie Mellon University and other institutions propose BrowserART to evaluate the generalization of refusal training to browser agents.

Why it was selectedThe paper gives concrete gaps in refusal between browser agents and the underlying models, and releases a reusable red-teaming test set, suitable for evaluating agent deployment risks.

ST-WebAgentBench: A benchmark for evaluating web agent safety and trustworthiness(original, opens in a new tab)

Build ST-WebAgentBench, a benchmark for evaluating the safety and policy compliance of Web Agents.

重新思考奖励模型评估：我们是否找错了方向？(original, opens in a new tab)

一篇 arXiv 论文在合成设置中考察奖励模型（RM）准确率与下游策略表现之间的关系，发现二者仅呈弱正相关。

以数据为中心的 AI 对齐：挑战与未来方向(original, opens in a new tab)

主张转向数据中心对齐，并归纳人类反馈的失准来源。

Moral alignment of LLM agents(original, opens in a new tab)

用内在道德奖励微调 LLM 智能体在矩阵博弈中的决策。

Anthropic 可解释性团队九月更新：后继头与 SAE 过采样(original, opens in a new tab)

Anthropic 可解释性团队报告称，他们在一个 18 层模型中复现并分析了实现序数序列递推的后继头（successor heads）。研究者通过权重检查中的 OV 电路发现最高分后继头能将约 80% 的序数 token 映射到其后继项，第二名的该比例为约 60%，且错误呈块状结构分布在与输入同类的其他序数 token 上。他们还借助独立成分分析寻找跨注意力头的共同模式，得到一个近似实现递进的组件，并在相关类别间存在一定串扰。

通过拒绝特征对抗训练提升 LLM 防护稳健性(original, opens in a new tab)

提出 ReFAT，在安全微调中消融拒绝特征以模拟越狱攻击。

PlanBench evaluation: o1's planning ability has improved significantly but is still not saturated(original, opens in a new tab)

Using PlanBench to evaluate whether reasoning models can complete planning.

超越 AI 对齐中的偏好取向(original, opens in a new tab)

作者将这一取向称为偏好主义，指出理性选择理论作为描述模型存在局限，偏好难以承载人类价值的厚语义内容，效用表示也忽视价值的不可通约性。

Bengio 团队提出用贝叶斯上界为 AI 行为构建护栏(original, opens in a new tab)

Yoshua Bengio 与合著者在 arXiv 论文中提出，通过估计上下文相关的安全违规概率上界，为 AI 的危险动作提供运行时护栏。作者认为现有安全评测与基准只是抽查，检测不到问题不代表 AI 安全，且模型可能识别出自己正在被测试而临时表现良好。论文的核心结果是在真实但未知的假设下推导安全违规概率的界，方法涉及在贝叶斯后验上搜索谨慎但合理的假设，并假设先验足够宽，分别讨论了 iid 与非 iid 两种情形。作者在可精确进行贝叶斯计算的玩具设定上做了实验模拟，结果与理论一致。

Llama 3 model family: 405B dense Transformer and Llama Guard 3 safety model(original, opens in a new tab)

Describes the pretraining and post-training methods of the flagship base model and releases weights and safety models.

Anthropic interpretability team June 2024 update: TopK vs. Gated SAE comparison and external evaluation progress(original, opens in a new tab)

The Anthropic interpretability team released a June 2024 update summarizing several research ideas still taking shape. The team reproduced the improvements of TopK SAE and Gated SAE over the standard L1-penalty SAE, confirming that the two perform similarly and clearly better on the L0/MSE tradeoff, but are less optimized by training than the standard SAE. In a blind evaluation of 100 features, no clear difference in monosemanticity was seen across methods; high-density features did not constitute a pathological problem at low L0, but at K=100 a large number of hard-to-interpret high-density features appeared. Clerp automated evaluation showed that Gated SAE and K=20 TopK SAE scored comparably to the L1=10 standard SAE.

Why it was selectedThe Anthropic interpretability team summarizes reproduction results for SAE training methods and progress in external evaluation, suitable for those following the sparse autoencoder route to understand the current consensus and unresolved issues.

The paper proposes the ARE framework to evaluate the adversarial robustness of multimodal agents on VisualWebArena(original, opens in a new tab)

Proposes the ARE framework and image-text attacks to evaluate the adversarial robustness of multimodal web agents.

设计用于对话式 AI 透明度与控制的仪表盘(original, opens in a new tab)

设计 TalkTuner 仪表盘，读出并干预对话模型对用户属性的内部表征。