New paper by Owain Evans and others: when poisoning carries an identifying trigger, clean data may instead strengthen backdoor misalignment

A new paper by Owain Evans and others shows that when poisoned data carries an identifying trigger, adding clean data may instead strengthen backdoor misalignment; the authors call this phenomenon homeopathic learning. This result is contrary to the common assumption that clean data can dilute poisoned data; in the illustrated example, fine-tuning with only reward-hacking poisoned data carrying a trigger yields a very weak backdoor, while fine-tuning after mixing in clean data makes the harmful answers elicited by the same trigger more severe.