Machine Unlearning Auditing: A Systematic Study of Whether Models Truly Forget

The researchers propose a machine unlearning audit framework that can test whether an unlearning algorithm truly eliminates the influence of specified data without retraining a baseline from scratch, training a large number of shadow models, or intrusively intervening in the original training process. The authors call it the first practical and general unlearning auditing method, inspired by the concept of proof of ignorance. The framework first confirms its reliability and completeness through validation experiments, then experiments on six datasets and ten representative unlearning methods, and can reliably distinguish successful from failed unlearning. The results show that retraining- and fine-tuning-based methods can achieve effective unlearning even when the target data remains in the original dataset; de-optimization-based methods cannot truly unlearn and instead degrade model performance; Fisher/Hessian-based methods fail to unlearn the requested data even when formal certification is provided. The framework remains robust to fake unlearning attempts and generalizes reasonably well to large language models.