Chunky Post-Training: Data-Driven Generalization Failures
Chunky Post-Training: Data Driven Failures of Generalization
Quick read
Based on 论文全文 （AI 生成）; refer to the original
The researchers propose the concept of chunky post-training, referring to models learning unintended correlations from post-training data that trigger behaviors such as refusal, contradiction, or code generation in inappropriate contexts, and they open-source two tools, SURF and TURF, for discovering and attributing such failures respectively. On Claude 4.5, GPT-5.1, Grok 4.1, Gemini 3, and the open-source model Tülu 3, this kind of behavioral misrouting is widespread and fairly robust to prompt rewording. TURF traces Tülu 3's failures to specific data, for example, about 250 samples rejecting made-up invention questions cause it to contradict real facts, and injecting LaTeX into math problems increases sympy hallucination calls by 50% and lowers accuracy. Data ablation shows that removing the relevant samples or adding correct examples mitigates the behavior.
- 后训练数据由多个针对特定行为的数据块拼成，会编码开发者未察觉的附带模式（格式与内容的关联、狭窄措辞、隐式联想），模型会忠实学到这些虚假关联，导致在普通输入上出现意外行为，如拒绝真实事实、把求助当谜题。作者称之为 chunky post-training，认为它影响用户信任、干扰评测，并暴露训练数据在教不想要的东西。
- 作者提出两个工具：SURF 是黑盒审计流程，给定失败模式的自然语言描述，在语义 prompt 属性上做两阶段搜索，用 LLM judge 按 rubric 打分并迭代加权属性池，自动找出违规的 query–response 对；TURF 则把违规行为追溯到训练数据，通过提取 query/response 属性、聚类 query 属性、检索相似 response 并统计命中，给出“当模型看到[trigger]就[behavior]”的归因。
- 在 Claude 4.5、GPT-5.1、Grok 4.1、Gemini 3 和开源 Tülu 3 上，SURF 发现 code、analytic、math、rebut、refusal 等行为在不当语境中被触发，rebut 和 refusal 得分最高，math 普遍较低；扰动重采样后失败率仍达数十个百分点。TURF 将 Tülu3 的多个行为归因到数据：正式词汇与代码数据相关、少量身份数据泛化到其他模型、LaTeX 触发幻觉 sympy 工具调用使数学准确率下降、数据抽取格式使逻辑推理准确率降 14%。数据消融显示移除或补充平衡数据可减少对真实事实的驳回。
- 研究只限单轮助手回复，多轮交互空间更大；归因只聚焦 SFT 数据，RL 等后训练阶段如何引入、抑制或重塑 chunky 行为仍待研究；数据消融未包含完整 RL 流程，RL 对 SFT 学到行为的影响方向不稳定、难以预测。未来可研究数据层面的缓解策略（删除、添加、增强样本），以及用 system prompt 等上下文抑制特定行为。
The paper attributes models' erroneous triggering of refusals, rebuttals, and other behaviors under specific phrasings to distributional imbalances in post-training data, and provides reusable tools for localization.