Leaner Training, Less Leakage: Revisiting Memorization in Large Language Model Fine-Tuning with LoRA

The paper by Fei Wang and Baochun Li (arXiv 2506.20856) revisits memorization in large language model fine-tuning and finds that LoRA fine-tuning has a markedly lower memorization risk than full fine-tuning while maintaining strong task performance. Memorization makes large language models vulnerable to data extraction attacks; memorization during pretraining has been widely studied, whereas memorization during fine-tuning, especially with the widely adopted parameter-efficient method LoRA fine-tuning, has been less explored previously. The study also finds that, contrary to previous conclusions, model scale and data repetition, two factors that strongly affect memorization in pretraining and full fine-tuning, do not follow the same trends in LoRA fine-tuning; the authors reach these conclusions using a more lenient similarity-based memorization metric.