Adversarial Robustness and Training of Unified Text Scoring Models
Unifying Adversarial Robustness and Training Across Text Scoring Models
Quick read
Based on 论文全文 （AI 生成）; refer to the original
Researchers at the University of Waterloo propose unifying the study of adversarial robustness of retrievers, rerankers, and reward models through text scoring, and provide several adversarial training methods. The framework defines an attack as making an irrelevant or unselected text score higher than a relevant or selected text, with failure conditions that can be directly checked. Experiments show that combining complementary adversarial training methods (Rudimentary, HotFlip, PGD, and content injection) is generally more robust than any single method, without sacrificing task performance. In RLHF, reward models trained with combined adversarial training are harder for policy models to exploit, reducing reward hacking, and the resulting policies are more preferred in LLM judgments.
- 语言模型对抗鲁棒性研究分散在不同应用与攻击上，掩盖了共享的脆弱性。作者主张用「文本评分」统一 dense retriever、reranker 和 reward model：攻击成功与否有清晰可测的判据——无关或未被选中的文本不应超过相关或被选中的文本。
- 作者把攻击统一为分数操纵，覆盖 rudimentary 扰动、HotFlip 梯度 token 替换、MLM 引导替换，以及此前未被处理的 content injection（句子注入、query 注入）。训练侧比较 Rudimentary、HotFlip、PGD、Injection、Paraphrasing 及组合策略，并在 RLHF 中检验对抗训练过的 reward model。
- 随着 Rudimentary/HotFlip 训练强度增加，对扰动类攻击的鲁棒性一致提升，PGD 类似但较弱；但针对注入与针对扰动/替换的鲁棒性转移很差。Content injection 训练最能提升注入鲁棒性，paraphrasing 收益有限。组合训练通常最鲁棒且常能同时改善下游效果。RLHF 中，用组合对抗训练 reward model 训练的 policy 比用 base reward model 更受 LLM judge 偏好、KL 漂移更小。
- 研究限于 110M–3B 的评分模型骨架，RLHF 最多到 8B，未直接研究生成式 LLM 的对抗训练；攻击覆盖有限，未涵盖全部已知攻击（如其他替换机制）。作者建议扩展到更大模型、把评分框架与生成式安全数据方法结合，并继续寻找广谱强鲁棒性的训练方法，因为没有方法对任一攻击达到完美鲁棒。
The paper unifies the adversarial robustness of retrievers, rerankers, and reward models as a text scoring problem, and provides a combination of adversarial training methods that transfer across attacks.