Skip to content
Original

Privacy Side-Channel Attacks in Machine Learning Systems

Privacy Side Channels in Machine Learning Systems

arXiv· arXiv:2309.05610· Edoardo Debenedetti, Giorgio Severi, Nicholas Carlini, Christopher A. Choquette-Choo, Matthew Jagielski, Milad Nasr, Eric Wallace, Florian TramèrETH Zurich; Northeastern University; Google DeepMind; UC BerkeleyIndexed by this site · 16 days ago· Originally publishedSelected AttacksSelectedAttention39

Translation in progress — untranslated text is omitted.

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

AI summaryResearchers from ETH Zurich, Google DeepMind, and other institutions propose privacy side-channel attacks against machine learning systems that can leak training data or user queries at rates far exceeding those of the model alone. The attacks cover four types of components: training data filtering, input preprocessing, output filtering, and query filtering. Under the exact deduplication setting on CIFAR-10, poisoning-combined membership inference reaches a 98% true positive rate at a 0.01% false positive rate, and deduplicating before differential privacy training invalidates the provable privacy guarantee. The authors use the memory filtering side channel of GitHub Copilot to infer that its training data collection cutoff is October 2021, and in experiments with GPT-Neo simulating memory filtering, they extract OpenSSH private keys from the training set with about a 90% success rate. The attacks can also extract GPT-2's full vocabulary, and the query filtering side channel can leak other users' test queries; on this basis, the authors argue for end-to-end system-level analysis of machine learning privacy.

Questions
Most existing ML privacy protection methods assume the model exists in isolation, but real systems contain components such as training data filtering, input preprocessing, and output monitoring. The authors argue that these system-level components can be exploited to leak privacy at rates far higher than from isolated models.
Methods
The authors propose the concept of privacy side channels and present four classes of attacks covering the full ML lifecycle: training data filtering (deduplication, poisoning defenses), input preprocessing (tokenizer truncation), output filtering (memory filtering), and query filtering (stateful detection), turning data dependencies introduced by components into membership inference, attribute inference, and data extraction attacks.
Experiments and results
On CIFAR-10, the attack reaches 99.6% TPR @ 0.1% FPR under exact delete-all deduplication; it infers that GitHub Copilot's training data cutoff is October 2021; it extracts about 90% of OpenSSH private keys from GPT-Neo with a memory filter; combining deduplication with DP-SGD makes the empirical ε at least 22 when the theoretical ε=3.
Limitations and future work
The authors point out: no technique yet exists that can deduplicate without exacerbating privacy leakage; how to enforce differential privacy under combinations of hard-to-analyze modules such as filters is an open problem; this paper focuses only on privacy side channels, and side channels for other types of attacks are left to future work. Experiments cover CIFAR-10, LFW, GPT-2/Neo and Copilot, among others, and some attacks assume the attacker can poison the training set.
Why it was selectedThe paper shows that common system components such as deduplication and output filtering amplify privacy leakage and can even invalidate differential privacy guarantees, which deployment teams can use to re-examine end-to-end privacy.

The paper shows that common system components such as deduplication and output filtering amplify privacy leakage and can even invalidate differential privacy guarantees, which deployment teams can use to re-examine end-to-end privacy.

Related events and news

0 events · 1 news mentions

News that mentions this paper

Related papers

Both use: CIFAR-10
Show 3 more papers
Both use: CIFAR-10
Read originalarxiv.org