Skip to content
Original

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

arXiv· arXiv:1802.08232· Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, Dawn SongGoogle Brain; University of California, Berkeley; National University of SingaporeIndexed by this site · 14 days ago· Originally publishedSelected ToolsSelectedAttention45

Translation in progress — untranslated text is omitted.

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

AI summaryThe paper proposes a test method for evaluating whether generative sequence models unintentionally memorize rare sequences in the training data, and provides a quantifiable exposure metric. The method inserts randomly generated canary sequences into the training data different numbers of times, and computes exposure by comparing their perplexity difference with sequences not inserted. The authors also give an extraction algorithm based on shortest-path search, extract credit card numbers and social security numbers from a language model trained on Enron email data, and measure and limit the exposure of rare sequences on Google Smart Compose. Experiments show that regularization methods such as weight decay, dropout and quantization cannot prevent this memorization, and that differentially private training can completely eliminate the problem but incurs some utility loss.

Questions
Generative sequence models are often trained on sensitive data and may unintentionally memorize and leak rare secret sequences in the training data (e.g., credit card numbers), but previously there was no method to quantify the degree of this memorization.
Methods
The authors propose the exposure metric: insert random canary sequences into the training data and quantify the degree of memorization using their relative perplexity rank; they also propose a Dijkstra-based shortest-path search algorithm to efficiently extract memorized secrets.
Experiments and results
Memorization appears early in training and is unrelated to overfitting; in the Enron email model, a real credit card number reaches an exposure of 52 and can be extracted; regularization cannot prevent it, and only differential privacy eliminates memorization.
Limitations and future work
The authors note that the method applies only to generative models, that the extraction algorithm assumes known context and is not suitable for real attacks, and that it uses only model input-output behavior. Experiments cover datasets including PTB, WikiText-103, and Enron, and architectures including LSTM, GRU, and CNN.
Why it was selectedThe paper proposes a quantifiable exposure metric and extraction algorithm, and uses Smart Compose and Enron email data to demonstrate generative models' memorization of rare training sequences.

The paper proposes a quantifiable exposure metric and extraction algorithm, and uses Smart Compose and Enron email data to demonstrate generative models' memorization of rare training sequences.

Related events and news

0 events · 1 news mentions

News that mentions this paper

Related papers

Both use: WikiText-103
Read originalarxiv.org