Analyzing personally identifiable information leakage in language models

1. 01What problem does this paper attempt to solve?

   How to quantify PII leakage of fine-tuned language models end-to-end, and how much the two defenses, scrubbing and DP, can actually block.

   Core question: how much personally identifiable information (PII) in training data a language model leaks after fine-tuning, and to what extent existing scrubbing and differential privacy (DP) defenses each mitigate it, currently lacks an end-to-end quantitative method.

   • Scenario and motivation: The authors use a next-word prediction service similar to Gmail Smart Compose as an example—an LM is fine-tuned on private data containing PII (emails, medical records, legal documents) and deployed via a black-box API, and an attacker may learn PII such as names and addresses; the authors believe prior work focused on membership inference and general data extraction and paid insufficient attention to PII leakage.
   • Shortcomings of existing defenses (as stated in the paper): PII scrubbing relies on NER, whose recall is imperfect (the paper cites 97% for names and 80% for nursing unit numbers on clinical data), and it must trade off disclosure risk against data utility; DP provides record/user-level guarantees, but the authors believe it assumes records are not repeated, whereas PII often repeats across records and across users, so the contribution of record-level DP to PII disclosure is unclear.
   • Core observation: The authors believe that "protecting by whom (who contributed the data)" is not equal to "protecting about whom (who the data is about)"; when the same PII appears repeatedly in multiple records, a DP model may still leak that information itself.
   • What the paper proposes: Game-based definitions and computable approximate attack algorithms for three PII leakage attacks (extraction, reconstruction, inference), and an empirical privacy/utility evaluation on three domains and four GPT-2 sizes.
   • Threat model:
     • Goal: Extract real PII from the LM (extraction), or associate PII with its context (reconstruction, inference).
     • Knowledge: The attacker knows the training pipeline and data distribution; reconstruction additionally receives cleaned training sentences with [MASK], and inference additionally receives a candidate set containing the correct PII (Table I).
     • Capability: Black-box API access, allowing queries of the full probability vector of the next token for any prefix; the paper mentions that the API is billed per token, which in practice limits the number of queries.
     • Victim system: A generative LM fine-tuned on private data and deployed via a black-box API (GPT-2 in the experiments).

   Collapse this answer

2. 02What related research exists?

   Related work falls into two categories: training data extraction and privacy defenses; the baseline directly compared in this paper is the TAB reconstruction attack.

   Training data extraction and memorization

   • Carlini et al. [9, 11] and others—study LM memorization and extraction of training data, including sentence-level extraction, canary extraction [8, 50, 68], and n-gram-level copying [42]; the authors note that such work does not distinguish public from private information and is mostly evaluated in undefended settings.
   • PII leakage research—PII leakage on masked language models [36, 61], large LMs [26, 54], and the Smart Reply classification model [32]; the authors note that these works mainly understand leakage without mitigation measures.
   • Causes of memorization—Jagielski et al. [31] study factors such as training data order, and Tirumala et al. [60] study the effect of dataset size, learning rate, and model size on memorization.

   Privacy defenses

   • DP fine-tuning—Yu et al. [67] and Li et al. [38] propose methods for differentially private fine-tuning of LMs on private data; Shi et al. [57] propose selective DP that adds noise only to sensitive samples.
   • Combined defenses—Stock et al. [59] study canary extraction and Rényi DP defenses for fine-tuned GPT-2 models; Zhao et al. [70] propose combining deduplication, editing, and DP-SGD, and the authors say "most interestingly" they use this paper's risks and metrics to evaluate the effectiveness of that scheme.

   Membership inference and its link to PII

   • Yeom et al. [65]—link attribute inference with membership inference (MI); this paper borrows this linking approach to connect PII reconstruction with record-level MI, and uses shadow-model MI attacks [58] for the empirical part.

   Baselines and benchmarks

   • Baseline method: the TAB attack [28] (greedily decoding the next span of tokens from the LM using a query prefix to reconstruct PII), directly compared with this paper's reconstruction attack on three datasets in Table IV.
   • Benchmarks/datasets: ECHR (European Court of Human Rights cases) [12], Enron emails [34], and a subset of Yelp-Health reviews; the authors also mention that the Text Anonymization Benchmark [51] is too small (1,268 records) to be used for DP-SGD fine-tuning.

   The authors' positioning of this paper: The authors state that this paper is the first to evaluate the interaction between PII scrubbing and DP in an end-to-end training pipeline, and that it focuses on PII leakage on fine-tuned LMs given context (prefix+suffix), without context, or under a candidate list, rather than on arbitrary types of memorization.

   Collapse this answer

3. 03How does the paper address this problem?

   Give game-theoretic definitions for three PII attacks, approximating the intractable exact leakage formula as executable black-box attack algorithms.

   Overall approach: first establish a taxonomy of PII leakage (extraction, reconstruction, inference) and game-based definitions, prove that exact leakage measurement is intractable, then give concrete attack algorithms that approximate these measures using black-box queries, and use them to quantify leakage under each defense. The method has no unified name; the core algorithms are Algorithm 3/4 (extraction) and Algorithm 6 (reconstruction/inference).

   Problem formalization and definitions

   • Extraction success rate: defined as the expected recall of the attacker's output PII set against the true PII set in the training set, Succ_EXTRACTION = E[|C ∩ C̃| / |C|]; advantage is defined as the difference from the success rate upper bound of an attacker without model access.
   • Extractability: EXTRACTABILITY(C; θ) = ∑_S Pr(S; θ) Pr(C|S; θ), i.e., the expected probability that the PII sequence C appears in samples generated by the model; the authors note that this expression sums over all sequences and is intractable.
   • Reconstruction objective: given a prefix S0 and a suffix S1, find arg max Pr(S0 C S1; θ); the authors note that this is an instance of constrained beam search, and exact solution is intractable.

   PII extraction (Algorithm 3, 4)

   • Observed leakage (Algorithm 3): sample N sentences from an empty prompt, use NER to mark PII of the same class as the target, and count the frequency of the target PII.
   • Lazy estimation (Algorithm 4): based on two assumptions—PII follows grammatical rules and PII of the same class is interchangeable—after sampling, replace with same-class PII and accumulate the probability the model assigns to the target PII, estimating extractability pointwise and avoiding running NER on massive generated text.

   PII reconstruction and inference (Algorithm 6)

   • Pipeline: 1. use a public masked language model (implemented as RoBERTa) to fill the remaining [MASK] in the query; 2. for reconstruction, sample N sentences from the target model with top-k and collect candidate PII with NER; 3. fill each candidate into the target position, rank by whole-sentence perplexity, and return the one with the lowest perplexity.
   • Inference: shares Algorithm 6 with reconstruction, but the candidate set C is known to the attacker (assumed to contain the correct PII); the authors observe that an attacker who only needs to infer from candidates is markedly stronger.
   • Baseline leakage handling: PII that the public model before fine-tuning could already extract or already predict correctly is excluded, and the authors believe the remaining leakage can be attributed to the model's observation of private data; they also acknowledge that this undercounts some sensitive leakage (such as health information of public figures).

   Evaluation method and key parameters

   • Data and annotation: three datasets (ECHR, Enron, Yelp-Health), with PII annotated by two NER tools, Flair (trained on OntoNotes, F1 90.93) and Presidio (based on spaCy), covering 21 entity classes in total (Appendix A); cleaning replaces PII with [MASK] (Algorithm 1).
   • Training settings: four GPT-2 sizes, Small/Medium/Large/XL, each trained in four variants: undefended, DP, scrubbed, and DP+scrubbed; DP uses the dp-transformers library, with per-sample gradient norm clipped at 1.0, training for 4 epochs, (ε, δ) = (8, 1/N).
   • Metrics: utility is test-set perplexity; membership inference is ROC AUC; PII extraction is Precision/Recall; reconstruction and inference are top-1 accuracy (Section IV-D).

   Collapse this answer

4. 04What experiments did the paper conduct?

   Three attacks are evaluated on three datasets and four GPT-2 sizes; DP substantially reduces leakage but does not eliminate it, and cleaning plus DP blocks PII attacks but harms utility.

   Experimental Setup

   • Models tested: GPT-2 Small (124m), Medium (355m), Large (774m), XL (1557m), each with undefended, DP(ε=8), scrubbed, and DP+scrubbed variants.
   • Datasets: ECHR (118,161 records, 16,133 unique person-class PII), Enron (138,919 records, 105,880), Yelp-Health (78,794 records, 17,035) (Table VII).
   • Baselines: reconstruction attack compared with the TAB attack [28]; membership inference uses a shadow-model attack [58].
   • Metrics and criteria: extraction uses Precision/Recall (PII in NER-labeled generated text); reconstruction/inference uses top-1 accuracy; utility uses test perplexity; MI uses ROC AUC.
   • Sampling setup: in extraction experiments, each target model is sampled via 15k queries for about 4m tokens (256 tokens generated per query, top-k sampling with k=40); baseline leakage is excluded using a sample of 50k queries and 13m tokens from public models.
   • Reconstruction/inference setup: reconstruction samples 64 candidates with top-k sampling k=40; inference is evaluated on GPT-2-Large with candidate set sizes |C| of 100 and 500.

   Main Results

   According to Table IV (top-1 accuracy of person-class PII reconstruction, this paper's attack with |C|=64):

   Swipe horizontally to read the table

   Dataset / Attack	GPT-2 Small No DP	GPT-2 Medium No DP	GPT-2 Large No DP	GPT-2 XL No DP	GPT-2 Large ε=8
   ECHR (TAB)	0.78%	1.21%	5.81%	4.30%	0.48%
   ECHR (Ours)	2.25%	3.36%	18.27%	13.11%	0.55%
   Enron (TAB)	0.59%	0.67%	1.75%	2.19%	0.04%
   Enron (Ours)	6.29%	7.26%	12.68%	15.25%	0.55%
   Yelp-Health (TAB)	0.33%	0.37%	0.65%	1.99%	0.12%
   Yelp-Health (Ours)	0.42%	1.31%	1.69%	6.40%	0.35%

   • Authors' interpretation: the authors state that this paper's attack is at least 2.5 times higher than TAB on ECHR, at least 7.5 times on Enron, and about 3 times on Yelp-Health, with a maximum of 10 times (Enron GPT-2-Medium); the exception is Yelp-Health on GPT-2-Small, which rises only from 0.33% to 0.42%.
   • Reconstruction under DP: reconstruction risk in DP models is ≤1%; the authors state that the attack still outperforms the baseline in all cases but consider the leakage too small to constitute a practical attack.
   • Model scale: the authors observe that larger models are more susceptible to PII reconstruction on all datasets.

   PII Extraction Experiments

   • Results (Table III): on ECHR, GPT-2-Large undefended achieves 22.96% recall and 29.56% precision; after DP(ε=8), recall is about 3% and precision about 3% across scales (e.g., Large has 2.98% recall and 2.92% precision). The authors accordingly state that DP reduces leakage by about an order of magnitude but still leaks about 3% of PII.
   • Risk factors: Fig. 5a shows that the number of repetitions of PII in the training set has a linear relationship with leakage frequency (the authors note this differs from the superlinear effect observed by Kandpal et al. [33] on models trained from scratch); Fig. 5b shows that DP models do not leak long token sequences, and in the 3–6 token range DP leaks about an order of magnitude less than undefended.
   • Other entity classes: undefended models can extract 14.1% of legal case numbers and 16.3% of organization names; DP models reduce these to 2.8% and 4.1%; in Enron's DP models, no real phone number leakage was observed, but email address leakage occurred (Fig. 6b).

   PII Inference and Membership Inference

   • Inference results (Table V, GPT-2-Large): undefended accuracy at |C|=100 is 70.11% for ECHR, 50.50% for Enron, and 28.31% for Yelp-Health; after DP(ε=8) these drop to 8.32%, 3.78%, and 4.29%. At |C|=500, ECHR is 51.03% undefended and 3.71% with DP. The authors consider that DP cannot fully defend against knowledgeable attackers in practical settings.
   • Membership inference (Fig. 7a, GPT-2-Large, ECHR): the shadow-model attack yields AUCs of 0.96, 0.82, and 0.505 for undefended, scrubbed, and DP+scrubbed models, respectively; the authors state that scrubbing mitigates MI but far less than DP.
   • Association between MI and PII: Fig. 7c shows that a sentence's memorization score is positively correlated with PII reconstruction success rate; Fig. 7b shows that samples generated from empty prompts have memorization scores near zero, which the authors state makes them of little use for MI.

   Other Ablations and Analyses

   • Privacy/utility summary (Table VI, GPT-2-Large, ECHR): undefended perplexity 14 (9 after masking generated PII), DP 14, scrubbed 16, DP+scrub 16; extraction, reconstruction, and inference metrics for scrubbed and DP+scrub are all 0% or 1%, with MI AUC 0.5.
   • Sampling volume (Fig. 5c, ECHR): recall rises with sampling volume to 23%, and precision falls from 50% at 500k tokens to 30% at 4m tokens.
   • Context length (Fig. 6c): the authors state that inference accuracy declines slightly in very large contexts, possibly because the expected memorization per token is lower in long samples.
   • Candidate coverage: the authors report that when the correct PII is in the candidate set, about 70% can be predicted correctly, but reconstruction attacks often fail to sample the correct candidate, limiting the success rate.
   • Estimating extractability (Fig. 6a): the lazy estimate correlates with observed leakage, but there are false-negative outliers (actually generated but estimated as non-extractable).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list limitations including NER dependence, attack coverage, and lack of benchmarks, and propose future directions such as DP-aware cleaning.

   Limitations and future directions noted by the authors

   • Lower bound due to NER dependence: Cleaning and leakage measurement use the same NER model, and tools make errors, so "our results provide a lower bound on leakage" (Section V, Need for Better Benchmarks).
   • Need to extend to other models and data: The authors write that "further research is needed to extend our findings to other LMs and datasets" (Section V, Limitations).
   • High-precision low-recall attacks not considered: This paper uses average-case metrics and does not evaluate high-precision/low-recall attacks; the authors state that their impact needs further study (Section V).
   • Lack of annotated benchmarks: The authors state that benchmark datasets for measuring PII leakage are lacking, and that the Text Anonymization Benchmark is too small (1,268 entries) to fine-tune with DP-SGD while balancing privacy and utility (Section V).
   • DP-aware cleaning: The authors suggest designing a heuristic cleaner that accounts for partial protection by DP and only cleans unprotected PII, to obtain a better privacy/utility trade-off (Section V, Conclusion).
   • Other extensions: The authors mention extension to syntactic/semantic similarity matching, linkability attacks that exploit correlations among PII, and optimizing other leakage metrics (Section V).

   Scope of the experiments

   • The models tested are GPT-2 Small/Medium/Large/XL, 4 sizes in total, all checkpoints fine-tuned after pretraining on WebText (Section IV-C).
   • The datasets are ECHR, Enron, and Yelp-Health, 3 English datasets in total; the main tables for reconstruction and inference report only the person entity class (Table IV).
   • The defense evaluation includes four training configurations: undefended, DP(ε=8), scrubbed, and DP+scrubbed (Section IV-C).
   • The inference attack is evaluated only on GPT-2-Large, with candidate sets of 100 and 500 (Table V).
   • The paper does not report an analysis of the effect of the attack's query budget on reconstruction/inference success rates, nor does it report human evaluation or agreement with human annotation.

   Collapse this answer

6. 06Summarize the main content of the paper

   Definitions and empirical results for three black-box PII attacks: reconstruction is up to 10 times higher than the baseline, and DP substantially reduces but does not eliminate PII leakage.

   An attack paper on measuring PII leakage in language models (IEEE S&P 2023), with authors from the University of Waterloo and Microsoft.

   • Problem: Fine-tuned LMs memorize PII in training data, but scrubbing is incomplete, the degree of protection DP provides against PII disclosure is unclear, and there are no end-to-end quantitative metrics.
   • Method: Establish game-based definitions for three black-box attacks: PII extraction, reconstruction, and inference; exact leakage is intractable, so approximation algorithms are given—extraction relies on sampling plus NER statistics, reconstruction uses query suffixes and a public masked model to generate candidates and ranks them by perplexity, and inference ranks over a known candidate set.
   • Reconstruction results: On ECHR, Enron, and Yelp-Health, this paper's attack achieves top-1 accuracy of up to 18.27% (ECHR, GPT-2-Large, undefended), which the authors say is up to 10 times higher than the TAB baseline (Table IV).
   • Extraction results: On ECHR, GPT-2-Large undefended achieves 22.96% recall and 29.56% precision; after DP(ε=8), recall is about 3% across sizes (Table III), which the authors say means DP reduces it by about an order of magnitude but does not eliminate it.
   • Inference results: GPT-2-Large undefended achieves 70.11% accuracy with 100 candidates (ECHR), dropping to 8.32% after DP (Table V); the authors believe DP cannot fully defend against an informed attacker in practical settings.
   • Conclusions and implications (in the authors' voice): The authors believe record-level DP largely limits PII leakage but cannot eliminate it alone, and that scrubbing remains a necessary part of the training pipeline; they advocate designing gentler cleaners that account for DP protection, and find that sentence-level membership inference correlates with PII reconstruction success rate.

   Collapse this answer

Related papers