Debugging differential privacy: A case study in privacy auditing
Debugging Differential Privacy: A Case Study for Privacy Auditing
Translation in progress — untranslated text is omitted.
An arXiv paper (2202.12219) by Florian Tramer and five other researchers shows that privacy auditing, in addition to estimating lower bounds on the privacy loss of differentially private algorithms, can also be used to find flaws in schemes that claim to satisfy differential privacy. As a case study, the authors audit a recently open-sourced implementation of a differentially private deep learning algorithm, finding with 99.99999999% confidence that the implementation does not satisfy its claimed differential privacy guarantee.
- A proof of differential privacy does not mean the implementation is error-free; errors in implementations or proofs are common and hard to detect. The authors use a real open-source DP training implementation as a case study to test whether privacy auditing can find such errors.
- Privacy audit of the official implementation of Backpropagation Clipping: train 100,000 models each on MNIST and on MNIST' with one poisoning sample added, use a loss-threshold-based membership inference attack to distinguish the two datasets, then use Clopper-Pearson confidence intervals to convert TPR/FPR into an empirical lower bound on ε.
- The attack's TPR is 4.922% and FPR is 0.174% (Figure 1), at p < 10⁻¹⁰ 下推出 ε > 2.79, 10 times higher than the claimed ε = 0.21. The authors located that the implementation divided each layer's gradient sensitivity by batch size an extra time, making the added noise too small; the original authors have acknowledged this and plan to fix it by increasing the noise.
- The paper is a single-system case study and does not discuss the limitations of the method itself. The experiments cover only one implementation and one set of hyperparameters on MNIST; the choice of poisoning sample and threshold is heuristic, and the authors state it has no theoretical basis.
Related papers
Both use: MNIST
Show 3 more papers
Both use: MNIST