Reconstructing Personally Identifiable Information from Proprietary Data in Supervised Fine-Tuned Models

The researchers propose a coverage-aware decoding algorithm, COVA, which under prefix-based attacks targetedly reconstructs specific users' personally identifiable information (PII) from large language models supervised fine-tuned on proprietary data, with reconstruction performance superior to baseline decoding methods across multiple datasets. The study constructs multi-turn user-centric question-answering datasets in medical and legal scenarios with PII added, for realistic leakage evaluation; experiments show that an attacker with only a small amount of context about the target user can substantially improve reconstruction success rates. The authors point out that small-scale proprietary SFT datasets cause models to learn target-user PII associations during fine-tuning that can be reconstructed, thereby causing privacy leakage of practical significance.