Skip to content

Translation in progress Some entries still include text in the original language.

Reward hacking · News

Oct 12 · Monday

Reward hacking · News

Latest publications and developments
Oct 9Fri
  1. Goodfire Research, BasetenGoodfire Research · 2 sources · 2 reports · 31

    Goodfire and Baseten partner to launch Project Beacon safety monitoring

    Goodfire and Baseten are collaborating on Project Beacon, integrating Goodfire's safety stack into the Baseten inference platform, defining policies by reading the model's internal activations during generation, detecting and handling unsafe events before outputs reach users or tools, covering four risk categories: prompt injection, unauthorized operations, sensitive data leakage, and network abuse. Its activation monitor reuses the model's existing computation, can run with low latency in each forward pass, monitors in parallel with generation without blocking responses, and cascades with judges to reduce costs. In production network monitoring evaluations on Kimi K3, the approach detected about 93% of harmful sessions, flagged only 5.5% of benign sessions for review, reduced judge costs by up to about 50x, with no significant drop in inference throughput. Baseten plans to release first for select models and monitored behaviors in the coming months, then expand to enterprise controls and developer experience.

Sep 10Thu
  1. Cisco · 36

    Cisco 发布 AI 安全与安全框架 v2,新增智能体自主性失效分类

    Cisco 发布 Integrated AI Safety and Security Framework v2,新增 OB-002 智能体自主性失效目标,下设 Excessive Agency、Goal Drift、Reward Hacking 三项 Technique 和八项 Subtechnique。v2 同时把 v1 中分开的 OB-001 Goal Hijacking(提示注入)与 OB-002 Jailbreak 合并为一个目标,理由是二者在实战中难以区分、所用对抗技术高度重叠,底层 Technique 仍保留输入来源与防御方式的差异。框架还配套编写了按类别划分的 constitution,用于界定范围、区分相邻类别,并按完整轨迹而非单条消息判断行为。Cisco 建议部署方采用最小权限工具访问、关键操作审批门、不可变任务与停止条件、独立结果验证和完整审计轨迹。

You have reached the end