Beyond Indistinguishability: Measuring Data Extraction Risk in LLM APIs
Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs
Quick read
Based on the paper's full text (AI-generated); refer to the original
Researchers propose a new framework for measuring data extraction risk in LLM APIs, arguing that indistinguishability metrics such as differential privacy bounds and low membership inference risk are neither sufficient nor necessary to prevent training data extraction. The paper proves that indistinguishability and non-extractability are mutually irreducible, proposes a definition of (l,b)-non-extractability requiring any black-box adversary to need at least 2^b expected queries for the API to output a protected l-gram substring, and gives a worst-case extraction risk upper bound estimator based on token ranking. Experiments cover GPT-2-Small and Llama-3.1-8B (Pile, BookSum) fine-tuned on Enron emails, showing that uncalibrated MIA signals (Loss, MinK) correlate more strongly with extraction risk than calibrated signals (Zlib, Ref-Loss), that naive deduplication may increase worst-case extraction risk, and that DP training yields limited benefit in reducing extraction risk at smaller privacy budgets ε. Code is open-sourced at https://github.com/Emory-AIMS/Inextractability。
- Industry often treats differential privacy bounds or low membership inference risk as proxy metrics for a model's resistance to memorization and data extraction. The authors point out that such indistinguishability is neither a necessary nor a sufficient condition for extraction risk, and may cause the extraction risk of LLM APIs to be misjudged.
- The authors formalize an extraction game, prove that indistinguishability and inextractability cannot be reduced to each other, and propose the (l,b)-inextractability definition: any black-box attacker needs at least 2^b expected queries to make the API output a protected l-gram. Based on this definition, the authors use the rank of ground-truth tokens to construct a worst-case extraction risk upper bound estimator, and extend it to untargeted and approximate extraction.
- Experiments on GPT-2/Enron and LLaMA-3.1-8B/Pile, BookSum show: the correlation between indistinguishability and extraction risk strengthens as sequences get longer, and is stronger for uncalibrated MIA signals; the authors' estimator gives tighter or more consistent upper bounds than greedy generation and probability extraction methods; member samples are easier to extract than non-members, and extraction cost is lower for large models; instruction tuning plus ParaPO and API access restrictions can significantly reduce the extractable proportion, while DP training yields limited improvement at small ε.
- The authors' worst-case upper bound assumes the attacker knows the full text and can optimize decoding parameters position by position, which is difficult to achieve in reality, so there is a gap between the theoretical cost and the real attack cost. The relationship between DP noise and effective extraction defenses remains unclear, and the authors list it as future work; the neighborhood suppression of approximate extraction relies on a local log-Lipschitz assumption.
The paper argues that differential privacy and membership inference signals cannot constrain data extraction risk; teams relying on DP training or MIA auditing can use this to re-examine API privacy guarantees.