In a search question-answering task, researchers demonstrate that reference-answer feedback poisoning can cause an agent to self-generate backdoor rules.
Agent的参考答案反馈投毒
The paper "When Feedback Becomes a Backdoor" (arXiv 2611.09348) proposes reference-answer feedback poisoning: an attacker only needs to tamper with the reference answers in training tasks to induce a SkillOpt-based self-optimizing agent to generate persistent backdoor rules on its own, without modifying model parameters, the optimizer, or Skill files. In the SearchQA experiment, after 30% of the reference answers in 400 training samples were poisoned, all three random seeds successfully retained the backdoor; all 600 questions marked with [ZEPHYR-17] triggered it, with 0 false triggers on normal questions, and the post-poisoning normal accuracy of 84.5% was roughly comparable to the control's 85.5%. The carrier effect proposed in the paper shows that when backdoor rules are bundled with normal improvement rules, they can pass the clean validation set along with overall performance gains, and poisoning the validation reference answers constitutes another attack entry point. However, on the SpreadsheetBench spreadsheet agent, the natural attack failed on all three random seeds due to two obstacles: rule abstraction and modification budget.
The paper presents the success rate of reference-answer poisoning and the mechanism of the carrier effect, which teams deploying self-optimizing agents can use to inspect their feedback and validation pipelines.