Skip to content
Original

Scalably Extracting Training Data from (Production) Language Models

Scalable Extraction of Training Data from (Production) Language Models

arXiv· arXiv:2311.17035· Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, Katherine Lee·Indexed by this site · 16 days ago· Originally publishedSelected AttacksSelectedAttention47

Quick read

Based on the paper's full text (AI-generated); refer to the original

AI summaryResearchers from Google DeepMind, ETH Zurich and other institutions propose a scalable training data extraction attack, demonstrating that attackers can extract several GB of training data from open-source models such as Pythia and GPT-Neo, semi-open-source models such as LLaMA and Falcon, and ChatGPT. For gpt-3.5-turbo, the authors design a divergence attack that makes the model repeat a single word indefinitely to break out of chat mode, causing it to output training data 150 times more frequently than normal behavior; at a query cost of $200, they extract more than 10,000 samples that are verbatim identical to training data, including sensitive content such as personally identifiable information of dozens of individuals. The authors merge The Pile, RefinedWeb, RedPajama and Dolma into a 9TB auxiliary dataset to verify the extraction results, and find that the larger the model and the longer the training, the more memorized data can be extracted.

Questions
Large language models memorize training data, and attackers can extract potentially private training text by querying the model. Existing studies either manually verify a small number of samples only on small models, or measure only an upper bound on discoverable memorization; the gap between the two is huge, and the actual amount extractable is unknown.
Methods
The authors define "extractable memorization" (the ability to construct prompts that make the model output verbatim without knowing the training set), use random token prompts to generate outputs at scale, and build a 9TB auxiliary corpus with a suffix array for automatic matching and verification; for the aligned ChatGPT, they find that repeating a single token can cause it to "diverge" and revert to base language model behavior.
Experiments and results
Large amounts of training data were extracted from both open-source and semi-open models, with larger models leaking more easily; with about $200 of queries to ChatGPT, tens of thousands of verbatim memorized samples were extracted, including PII, code, URLs, etc.; the leakage rate of divergence attacks was far higher than that of normal conversation; alignment did not eliminate memorization, and ChatGPT's discoverable memorization rate was very low.
Limitations and future work
The auxiliary corpus is not a superset of the training set, so the results are only a lower bound; Good-Turing extrapolation severely underestimates when samples are insufficient; the divergence attack is effective only against gpt-3.5-turbo and is not general; aligned models are difficult to audit externally, and better extraction and evaluation methods are needed in the future.
Why it was selectedThe paper shows that with only $200 in query cost, tens of thousands of training data can be extracted from ChatGPT, indicating that current alignment techniques have not eliminated memorization.

The paper shows that with only $200 in query cost, tens of thousands of training data can be extracted from ChatGPT, indicating that current alignment techniques have not eliminated memorization.

Related events and news

2 events · 4 news mentions

News that mentions this paper

评论文章称语言本身已成为AI攻击面

  • Tech Policy Presscites the paper's id or link
    AI 时代，语言本身已成为攻击面
    jailbreaking LLMs in adversarial attacks for example, or semantic satiation causing a model to glitch and give up its PII dead.

AI训练数据集泄露数十万有效凭证

Show 1 more news items
Read originalarxiv.org