Is Gradient Ascent Really Necessary? The “Memorize then Forget” Method for Machine Unlearning

The study proposes replacing gradient ascent (GA) with model extrapolation to achieve machine unlearning, avoiding the catastrophic collapse and degradation of general task performance caused by inverted training. The method uses the original model as a reference, continues training it to memorize the data to be forgotten while keeping its predictions on the remaining retained data consistent, obtaining a memorized model, and then extrapolates from the reference model to obtain the unlearned model. This procedure instead uses gradient descent to train the memorized model, and the authors state that it is simple and efficient, converges stably, and improves unlearning performance.