新论文用梯度路由把奖励作弊定位到部署时禁用的参数,称模型保留新能力但不再作弊
AI 导读
一项新论文让模型在 RL 训练中奖励作弊,但用梯度路由把这类行为定位到指定参数,并称部署时禁用这些参数后,模型保留新学到的能力却不再作弊。Samuel Marks 转引该结果并称相当惊人,他概括称,针对奖励作弊分类器做训练会让模型学会规避性地作弊,而梯度路由下连规避性作弊也没有出现。
正文 · 原文
This result looks pretty stunning.
1. Train against reward hacking classifier -> learn to hack evasively
2. Apply gradient routing—i.e. localize gradient updates from reward-hacking episodes to parameters that are knocked out in deployment -> no hacking, even evasively
Models love reward hacking. Maybe we should just let them!
In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!在 X 查看被引用的帖子