Forget what follows: LLM unlearning based on statistical independence
Forget What Comes Next: LLM Unlearning via Statistical Independence
Translation in progress — untranslated text is omitted.
Researchers propose PINT, an LLM machine unlearning algorithm based on statistical independence that uses HSIC (Hilbert-Schmidt Independence Criterion) to reduce the statistical dependence between the tokens to be forgotten and the internal representations used to predict them. The paper's starting point is that poisoned training samples may still be statistically detectable after unlearning, which raises the question of whether existing unlearning objectives truly remove what the model has memorized; the authors formally relate the HSIC objective to Exact Memorization through a computable upper bound, directly linking the training objective to the model's ability to reproduce the forgotten tokens. Experiments compare the unlearning-utility trade-off across eight settings, with models up to 7B parameters, and demonstrate scalability to 70B-parameter models; the evaluation covers Gaussian poisoning, personal information, biosecurity and cybersecurity hazardous knowledge, and copyrighted books and news articles. The authors report that PINT achieves the strongest unlearning-utility trade-off on average, and call it the only method that can make Gaussian poisoning undetectable while maintaining utility close to the original model.
- LLMs memorize and reproduce training data, posing privacy and legal risks; the authors state that existing approximate unlearning methods may fail to remove poisoned samples, which remain statistically detectable after unlearning.
- PINT minimizes the HSIC between prefix representations and the next token on the forget set, constrains representations to stay close to the original model on the retain set; the authors also prove an HSIC upper bound for Exact Memorization.
- The authors compare the unlearning-utility trade-off on eight settings (up to 7B); PINT has an average area of 0.95, and the highest baseline is 0.90; on Gaussian poisons, PINT makes them undetectable and retains 91% utility; on the 70B model, WMDP-Cyber accuracy drops from 53.0% to 27.6%.
- The authors state that the evaluation is limited by existing benchmarks and that the metrics are imperfect proxies for unlearning; only text-only models are studied, and the impact on pre-deployment alignment (e.g., safety behavior) is unknown. The experiments cover seven benchmarks and six models, validated up to 70B.