Adversarial Targeted Forgetting in Regularization and Generative Based Continual Learning Models

Researchers Muhammad Umer and Robi Polikar propose an adversarial targeted forgetting attack against continual learning models, which injects backdoor samples as misleading information into the training data, forcing the model to learn and retain this false content, enabling the attacker to control the model at the test stage. The attack exploits the ability of continual learning algorithms to retain existing knowledge over the long term; building on prior perceptible misleading research against Elastic Weight Consolidation (EWC), it extends to other regularization-based and generative replay-based continual learning algorithms, and further extends to misleading samples imperceptible to the human eye. The authors evaluate this vulnerability on rotated and split benchmark variants of MNIST under both domain-incremental and class-incremental learning scenarios, showing that an attacker can create "false memories" for arbitrary tasks by inserting carefully designed backdoor samples, controlling the degree of forgetting for selected tasks. The authors state that the vulnerability is significantly harmful: backdoor samples need only be mixed into 1% of the training data to corrupt the model's memory, even when the misleading information is imperceptible to the human eye.