Privacy onion effect: memorization is relative

1. 01What problem does this paper attempt to solve?

   After deleting the training samples most vulnerable to membership inference attacks, are the originally safe samples safe? The authors state that this is not the case.

   Core question: Can deleting the outliers most vulnerable to privacy attacks from the training set give the remaining samples the expected privacy protection?

   • Problem setting: Deep models memorize training data, and privacy leakage risk is concentrated on a small number of outliers; a common practice in privacy auditing and machine unlearning is to identify and delete the most vulnerable samples.
   • Shortcomings of existing practice: The authors state that current empirical privacy auditing lacks "stability"—after a small amount of training data is deleted, users' empirical privacy risk may change substantially; performing unlearning based on audit results may harm other users' privacy.
   • Core observation (Privacy Onion Effect): The authors state that after deleting the most attackable "layer" of outliers and retraining, a layer of samples that were originally safe becomes exposed as new outliers and becomes vulnerable to the same attack. For example, deleting the 5,000 most vulnerable samples is mathematically expected to improve privacy by 15×, but in practice improves it by only 2×.
   • Paper contributions: Empirically demonstrate this effect on CIFAR-10/100, rule out four alternative explanations one by one—statistical noise, dataset shrinkage, limited model capacity, and duplicate samples—and propose the PRIVINF score to locate the few samples that "mask" target samples.
   • Threat model:
     • Attacker goal: membership inference, determining whether a given sample is in the model's training set.
     • Attacker knowledge and capability: train multiple shadow models and use LiRA to compare the Gaussian distribution likelihood of the target sample's logit-gap on the two classes of models.
     • Victim system: neural network classification models trained on CIFAR-10/100 (50,000 training samples each).

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: memorization and differential privacy, membership inference attacks, and privacy auditing and machine unlearning.

   Memorization and differential privacy

   • Feldman [11] proposes the definition of label memorization: a sample is memorized if deleting it significantly changes the probability that the model outputs its label; differential privacy (DP) [1, 10] provably prevents such memorization, but usually at the cost of accuracy [32].
   • Brown et al. [4], Feldman [11] et al. point out that on long-tailed distributions memorization is necessary for high-accuracy learning; Feldman & Zhang [12] et al. use influence estimation to identify memorized outliers.
   • The paper points out that DP provides a uniform guarantee for arbitrary datasets, whereas this paper studies: whether a privacy mechanism can be customized only for samples at risk (related to the instance-specific DP literature [25]), i.e., whether deleting easily memorized samples can substitute for a uniform constraint on the training algorithm.

   Membership inference attacks

   • Membership inference was proposed by Homer et al. [14] and introduced to the machine learning setting by Shokri et al. [29]; subsequent work provides prediction access at different granularities (from full confidence vectors to label-only [9]).
   • This paper adopts the Likelihood Ratio Attack (LiRA) of Carlini et al. [6], which the paper states achieves state-of-the-art on all metrics; ROC evaluation follows the best practices of that paper.

   Privacy auditing and machine unlearning

   • Jagielski et al. [16], Murakonda & Shokri [23] empirically audit privacy risk with concrete attacks; Tramèr et al. [33] show that adversarial additions to data can produce similar risk instability effects.
   • Machine unlearning [5] often uses membership inference as a metric for whether unlearning succeeded [3, 13]; Thudi et al. [31] have pointed out that membership inference metrics should not be relied upon to audit unlearning, and the results of this paper corroborate this finding.

   Positioning: The authors state that this paper's distinction is that it studies the alternative route of "modifying the training dataset" (deleting at-risk samples), and shows that such deletion-based defenses are unlikely to be effective.

   Collapse this answer

3. 03How does the paper address this problem?

   LiRA is used to compute a privacy score for each sample; the 5,000 most vulnerable samples are deleted, the model is retrained and re-measured, and controlled experiments are used to rule out alternative explanations.

   Overall approach: first quantify the privacy risk of each training sample, delete the highest-risk layer and retrain, then measure whether the risk of the remaining samples drops as expected, and finally use controlled experiments to locate the source of the effect.

   Problem setup and privacy scores

   • The dataset X is CIFAR-10/100 (50,000 training samples each); the subset distribution DX independently samples each sample with probability 50%, and the model is trained as fs ← T(Xs).
   • The privacy score of each sample is defined as the average attack success rate: ASR(x, X) := Pr[A(x, f_s) = 1[x ∈ X_s]], i.e., the probability that the attack correctly predicts x's membership (averaged over subset sampling and training randomness); attack advantage is defined as 2·ASR − 1.
   • The attack uses LiRA: multiple shadow models are trained, and the logit-gap of the target sample (the difference between the highest and second-highest logit) is modeled as a univariate Gaussian separately on models that trained on x and models that did not train on x, and the likelihood of the observed value is compared to decide membership.

   Three-step main experiment procedure

   1. Scoring: compute the average ASR of each sample over 200,000 models; using the open-source training pipeline [20], each model takes 16 GPU-seconds to train, using 16 A100s in total for 1000 GPU-hours.
   2. Deletion: sort by score and delete the 5,000 most attackable samples (10% for both datasets), yielding 45,000 "hard-to-attack" samples; the authors state these samples are usually outliers, consistent with the outliers identified by [12] (visualized in Appendix Figure 8).
   3. Re-measurement: retrain on the reduced dataset and rerun LiRA; the authors report that model accuracy on the reduced dataset is at most 2 percentage points lower than the baseline.

   Controlled experiments ruling out alternative explanations

   • Statistical noise: compute scores separately from two groups of 100,000 independent models each and compare their correlation.
   • Dataset becoming smaller: instead delete the 5,000 hardest-to-attack samples or 5,000 random samples, and compare ROC.
   • Model capacity: after deleting 5,000 CIFAR-10 outliers, add 5,000 randomly labeled CIFAR-100 outliers, to see whether arbitrary outliers produce the same effect.
   • Duplicate samples: use the imagededup library (CNN detection algorithm, threshold 0.85) to remove 5,275 duplicate images from CIFAR-10 and redo the experiment.

   Mechanism analysis: PRIVINF influence scores

   • Define PRIVINF(REMOVE(x) ⇒ x₀) as the expected membership inference accuracy for target x0 over all models that do not contain x (extended from the counterfactual influence of [36]), which can be computed simultaneously with LiRA using the same set of random splits.
   • For a target sample, delete the k points with the highest PRIVINF and observe the change in its attack advantage; evaluated separately on four types of targets: duplicate points, second-layer outliers, random points, and "safe points" with initial advantage below 2%.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The main experiment shows that the benefit of deleting the most vulnerable samples is only about 1/6 of the expected value; four control experiments rule out alternative explanations, and the PRIVINF experiment supports the "local masking" explanation.

   Experimental Setup

   • Target of evaluation: neural network classifiers trained on CIFAR-10 and CIFAR-100 (50,000 training samples each); the paper does not state the specific architecture names.
   • Attack: LiRA membership inference attack [6], training shadow models to model the logit-gap distribution.
   • Scale: in the main experiments, the ASR for each sample is averaged over 200,000 models; training used 16 A100s, totaling 1000 GPU-hours.
   • Metrics: ROC curves (TPR vs. FPR, y-axis on a log scale); specific values are read at FPR=0.01%; attack advantage = 2·ASR − 1.
   • The interpretability experiments (Sections 4 and 5) were conducted only on CIFAR-10 due to compute cost; number of repetitions: the noise check used two groups of 100,000 models each.

   Main Results

   Figure 1 shows three ROC curves for CIFAR-10 (left) and CIFAR-100 (right): baseline (blue), ideal (green, ignoring the 5,000 most attackable samples during the attack), and realistic (orange, actual deletion and retraining). Readings for CIFAR-10 at FPR=0.01%:

   Setting	TPR (FPR=0.01%)	Source
   Baseline (full 50,000 samples)	1.5%	Figure 1 / Section 3.3
   Ideal (restricting the attack to only the 45,000 safer samples)	0.1%	Figure 1 / Section 3.3
   Realistic (deleting the 5,000 most vulnerable samples and retraining)	0.6%	Figure 1 / Section 3.3

   • The authors state: under the ideal setting the attack success rate should drop to 1/15 (CIFAR-10) or 1/5 (CIFAR-100), and actual deletion is over 6× less effective than expected ("over 6× less effective").
   • Based on this, the authors propose the Onion Effect: deleting the current outliers merely makes a new batch of samples become outliers; the problem is shifted rather than solved.
   • Appendix A: iterative deletion (50 rounds, deleting the top-100 each round) and one-time deletion of 5,000 yield almost completely identical results, and the samples deleted by the two schemes overlap by more than 80% (Figure 7).

   Ruling Out Four Alternative Explanations (Section 4, CIFAR-10 only)

   • Statistical noise: the correlation coefficient of the privacy scores computed from two groups of 100,000 models each is r=0.9998 (Figure 2); of the 5,000 most attackable samples in the two runs, 4,974 are the same. The authors state that noise cannot explain the effect.
   • Dataset becoming smaller: after deleting the 5,000 hardest-to-attack samples or 5,000 random samples, the ROC is almost identical to the baseline (Figure 3). The authors state that the effect is not caused by shrinking the dataset by 10%.
   • Model capacity: after adding 5,000 randomly labeled CIFAR-100 outliers, the ROC of the remaining 45,000 samples shows no statistically significant difference from the ideal curve (Figure 4). The authors state that arbitrary outliers do not occupy capacity to produce this effect; only specific outliers matter.
   • Duplicate samples: after deduplication (deleting 5,275 duplicate images), the Onion Effect still holds (Figure 5); the authors also note that deduplication raises the attack success rate for some samples, because duplicate samples "mask" each other.

   PRIVINF Mechanistic Experiments (Section 5.1)

   • Duplicate points: for the 5 points whose ASR rises the most after deduplication, the highest PRIVINF is their duplicate samples; deleting these duplicate samples accounts for all of the attack improvement brought by deduplication, with membership inference advantage increasing by an average of 42 percentage points.
   • Second-layer outliers: for the 10 points whose advantage rises the most after deleting the 5,000 most vulnerable samples (no duplicates), deleting all 5,000 raises their advantage on average from 0.42 to 0.8 (+0.38); deleting only the 25 points with the highest PRIVINF for each already yields an increase of 0.22 (56%). The authors state this shows the Onion Effect is a local effect: each second-layer outlier is masked by only a few first-layer outliers (Figure 6 example: deleting 10 samples raises the target membership inference accuracy from 66% to 81%).
   • Random points and safe points: for 10 random points, deleting 25 high-PRIVINF samples raises the advantage only from 0.34 to 0.42 (+0.08); for 10 safe points with initial advantage below 2%, the average rises from 0.02 to 0.14 (+0.12), with the highest individual increase at 20 percentage points. The authors state that the effect is concentrated in the "second layer" and does not hold uniformly across the entire training set.

   Other Ablations and Analyses

   • Iterative deletion vs. one-time deletion: the ROC is almost identical, with a sample overlap rate of more than 80% (Appendix A, Figure 7).
   • Model accuracy on the reduced dataset: at most 2 percentage points lower than the full-data baseline (Section 3.2).
   • Appendix B (Figure 8): the most attackable samples visualized by class; the authors state they are consistent with the outlier/memorized samples identified by [12, 18].
   • Appendix C: deduplication used imagededup's CNN detection algorithm with a threshold of 0.85, deleting 5,275 duplicate images, and duplicate clusters were manually spot-checked.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors state that audit results are valid only for a specific dataset and that deletion-based defenses are ineffective; the explanatory experiments cover only CIFAR-10.

   Limitations and future directions noted by the authors

   • Audit conclusions cannot be extrapolated (Section 6): The authors state that the results of any single privacy audit should be limited to the exact dataset used when training the model, and cannot be regarded as an audit that remains stable after the dataset changes.
   • Deletion-based defenses are ineffective (Section 6): The authors state that the Onion Effect shows that defenses relying on deleting samples from the training set will be ineffective; the abstract further states that memorization-prevention proposals trained without strict privacy guarantees are unlikely to be effective.
   • Adding arbitrary OOD data is ineffective (Section 6, corresponding to the negative result in Section 4.3): The authors state that inserting more extreme (out-of-distribution) outliers does not make the original outliers more private.
   • Duplicating samples is instead harmful (Section 6): The authors argue that duplicating samples makes membership inference easier (distinguishing 0 copies from 2 copies is easier than distinguishing 0 copies from 1 copy), and cannot serve as a defense.
   • Open question (Section 6): The authors consider whether privacy can be improved by modifying the training dataset to be an interesting open question.
   • Risks of machine unlearning (Section 5.2): The authors state that membership inference metrics should not be used to audit unlearning; and raise the possibility of "adversarial unlearning"—an attacker can selectively cause others' data to be unlearned in order to maximize membership inference success against a target individual.

   Scope of experiments

   • The datasets are two image classification benchmarks, CIFAR-10 and CIFAR-100; the explanatory and mechanism experiments in Sections 4 and 5 were conducted only on CIFAR-10 (the paper states this is due to compute cost).
   • The attack evaluated only membership inference (LiRA); privacy scores are averaged over 200,000 models, and the noise check used two groups of 100,000 models each.
   • The PRIVINF mechanism experiments use small target sample sizes: 5 duplicate points, 10 second-layer outliers, 10 random points, and 10 safe points.
   • The paper does not report the specific architecture name or training hyperparameters of the neural networks used (only a training pipeline of 16 GPU-seconds per model).
   • The paper does not report specific TPR values for CIFAR-100 at FPR=0.01% (only the ideal factor of 5× and the ROC figure).

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors state that deleting the most vulnerable samples only transfers privacy risk to a new layer of samples, and that both deletion-based defenses and static auditing are unreliable.

   An analysis paper on training data memorization and privacy that proposes and empirically demonstrates the Privacy Onion Effect: after deleting the outliers most susceptible to membership inference attacks and retraining, samples that were originally safe become new outliers.

   • Problem: privacy leakage risk is concentrated on a few outliers, and a common practice is to audit and delete the most vulnerable samples; the authors ask whether this practice is actually effective.
   • Method: use the LiRA membership inference attack on 200,000 models to compute the average attack success rate for each training sample of CIFAR-10/100, delete the 5,000 most attackable (10%), retrain and re-measure, then use controlled experiments to rule out alternative explanations.
   • Main result: on CIFAR-10 at FPR=0.01%, the baseline TPR is 1.5%, the ideal expectation was a drop to 0.1%, but it actually only drops to 0.6%—the authors state the effect of deletion is more than 6 times worse than expected (Figure 1).
   • Explanations ruled out: statistical noise (scores computed from two groups of 100,000 models each, correlation r=0.9998), the dataset shrinking by 10%, model capacity being occupied by arbitrary outliers, and duplicate samples, all ruled out by controlled experiments (Figures 2–5).
   • Mechanism: PRIVINF influence scores show that deleting 25 high-influence samples achieves 56% of the advantage increase obtained by deleting all 5,000, and the authors state the effect is local—a few first-layer outliers mask second-layer samples; the effect on random points and safe points is much smaller.
   • Authors' conclusions: empirical privacy auditing holds only for specific (dataset, model) combinations and may be misleading after the data changes; machine unlearning may harm other users' privacy and can even be adversarially exploited; deletion-based defenses without strict privacy guarantees are unlikely to be effective, and whether privacy can be improved by modifying the dataset is an open question.

   Collapse this answer

Related papers