Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets

1. 01What problem does this paper attempt to solve?

   The authors propose a new class of attacks: poisoning training data to amplify privacy inference leakage about others' training samples.

   Core question: Can an attacker who can inject data into the training set thereby amplify privacy inference (membership inference, property inference, data extraction) about other parties' private training samples.

   • Problem setting and importance: The authors point out that the cryptographic principle "no integrity, no privacy" also applies to machine learning; training data for large models is hard to sanitize, and in decentralized settings (federated learning, secure multi-party computation) parties' data cannot be inspected.
   • Two independent lines of existing work: Data poisoning attacks target training integrity (degrading performance or implanting backdoors), while privacy attacks (membership inference, property inference, data extraction) target training data privacy; the authors state that no prior work studied the threat model of using poisoning to break other users' privacy.
   • Core observation: The most vulnerable to privacy attacks are outliers, because they have a large influence on the model; the attacker cannot modify the target sample, but can use poisoning to "turn" the target sample into an outlier, thereby amplifying the difference in model behavior between the two worlds of membership/non-membership.
   • Paper contributions: Proposes targeted poisoning attacks (<0.1% of data) that improve three types of privacy attacks by 1 to 2 orders of magnitude; untargeted attacks (controlling up to 50% of data) that reduce membership inference error rate on all other users' data by 8×; and gives attacks on kNN and linear SVM that provably achieve 100% membership inference accuracy (Appendix D).
   • Threat model:
     • Goal: Amplify privacy leakage about other parties' samples in training set D, winning the privacy inference game (guessing the target z).
     • Knowledge: Knows the data distribution D, the target value space U, and the training algorithm T.
     • Capability: Injects N_adv arbitrary poisoning points into the training set at once (targeted at minimum N_adv=1, untargeted at maximum N_adv=|D|), after which it can only query the trained model; cannot participate in the training process.
     • Victim system: Models trained on the poisoned dataset, including multi-party joint training settings such as MPC.

   Collapse this answer

2. 02What related research exists?

   Related work is divided into four groups: privacy attacks, integrity poisoning, privacy compromise under stronger threat models, and defenses and auditing.

   Training data privacy attacks

   • Shokri et al. [62] proposed membership inference; Fredrikson et al. [21, 22] and Yeom et al. [75] studied attribute inference; Carlini et al. [13, 14] studied extracting training data from language models. The authors state that these three are the most commonly considered training data privacy attacks in the literature, and this paper takes them as attack targets.
   • Carlini et al. [11] proposed membership inference based on per-sample likelihood ratios (calibrated with shadow models); this paper's MI attack directly extends that method; the attribute inference attack of Mehnaz et al. [45] is the baseline for Section 5 of this paper.

   Training integrity attacks (poisoning)

   • Indiscriminate (availability) attacks [7, 15, 20, 31, 51], targeted attacks [23, 61, 64], backdoor attacks [4, 6, 67], and supply-chain backdoors [26, 39, 40]. The authors point out that none of these poisoning attacks target privacy compromise.

   Stronger threat models where poisoning breaks privacy

   • An attacker who controls part of the training code can use the model as a side channel to steal training data [3, 63]; in federated learning, a malicious server can choose an architecture that facilitates sample reconstruction [9, 19]; participants in decentralized protocols can send dynamic malicious updates to amplify privacy attacks [28, 46, 53, 71]. The authors state that the difference between this paper and these works is that this paper only assumes the attacker can add a small amount of arbitrary data once and does not participate in training afterward.
   • Chase et al. [43] consider a threat model similar to this paper's, but with a weaker goal: inferring global properties of the training set (such as class distribution); this paper targets privacy leakage of individual samples.

   Defenses and privacy auditing

   • Poisoning defenses achieve robust training by detecting and removing out-of-distribution samples [15, 16, 27, 31, 66]; privacy defenses include heuristically reducing memorization [34, 52] and differentially private training [1, 17]; the authors state that differential privacy provably protects arbitrary (including poisoned) datasets.
   • Jagielski et al. [32] and Nasr et al. [54] use poisoning to measure the privacy guarantees of differentially private algorithms; the authors point out that this is fundamentally different from this paper: they measure the leakage of the poisoned samples themselves, whereas this paper shows that poisoning harms other benign samples.

   Positioning: The authors state that this paper introduces a new attack category, "active inference attacks," connecting the two independent threads of integrity and privacy, and relying only on the weakest data-injection assumption.

   Collapse this answer

3. 03How does the paper address this problem?

   Poison the target sample into an outlier to amplify its influence, then read out the leakage with a calibrated privacy attack.

   Overall idea: the attacker changes the training set through poisoning to maximize the difference in model behavior between the two worlds of "the target sample is in the training set" and "it is not"—that is, turning the target into an outlier, then running a (calibrated) privacy inference attack on the trained model. The paper instantiates this idea as three types of attacks: membership inference, property inference, and canary extraction.

   Formalization of the privacy inference game and threat model

   • Game 3.1 (standard privacy game): The challenger samples a dataset D and target z, trains a model on D∪{z}, the attacker queries the model and outputs a guess ẑ, winning if correct. Membership inference corresponds to U={x,⊥}, property inference to the various property versions of the target sample, and data extraction to all possible values of the secret.
   • Game 3.2 (the poisoning version in this paper): Before training, the attacker submits to the challenger a poisoning set D_adv of size N_adv, and the model is trained on D∪D_adv∪{z}. The attacker knows D, U, T, and the poisoning need not be stealthy.
   • Success metrics: Membership/property inference use TPR at low FPR (e.g., 0.1%); canary extraction allows multiple guesses and is measured by exposure (bits): exposure(z;Ẑ) = log₂|U| − log₂ rank(z;Ẑ), i.e., the information bits converted from the position of the correct secret in the attacker's ranking.

   Core principle and provable warm-up

   • Key distinction: The goal is not to make the model memorize the target sample (simply copying the target would memorize it in both worlds, which does not help distinguish), but to make the presence of the target sample have the greatest impact on model behavior.
   • kNN attack (Appendix D, Algorithm 1): Poison k points: k−1 copies of the target x (half labeled correctly, half labeled incorrectly) plus one mislabeled near neighbor at distance δ from x, making the target the deciding vote for classifying x; the model correctly classifies x if and only if the target is a member. The authors prove this achieves 100% membership inference accuracy, and that there exist "unused" points for which inference without poisoning is no better than random.
   • SVM attack (Appendix D): For "protruding" points on the convex hull, inserting poisoning points of a different class near them can force them to become support vectors; the authors prove (Theorem D.2) that membership can then be inferred with 100% accuracy via the distance to the decision boundary.

   Targeted attack design on deep networks

   • Label flipping (membership inference, Section 4): For target (x,y), insert r mislabeled copies (x,y′), r∈{1,2,4,8,16}, with y′ chosen randomly and consistent across copies. The model learns to associate x with y′, so the correctly labeled target becomes an outlier.
   • Property inference (Section 5): For the two versions x0, x1 of the unknown property, insert r/2 mislabeled copies of each, making the model's loss large for both versions; the presence of the true version leaves a detectable effect on one of the losses. The attack uses shadow models to fit a Gaussian distribution of the loss difference for a likelihood ratio test.
   • Canary extraction (Section 6): Two strategies—prefix poisoning (the attacker chooses the context before the canary, using out-of-distribution prefixes such as code or random tokens to increase exposure) and suffix poisoning (when the prefix is fixed, repeatedly insert the prefix plus an arbitrary suffix into the training set, "mislabeling" the token after the prefix).
   • Calibration: Use N shadow models to fit the loss distributions of members/non-members (or the various property versions); canary extraction uses calibrated confidence, i.e., the target model's log probability of "prefix+guess" minus the mean over N shadow models (Equation (3) in the paper).

   Untargeted attacks and preliminary exploration of defenses

   • Untargeted label flipping (Section 4.4): The attacker samples data from the same distribution in an amount equal to the victim's, all mislabeled as the same random class, turning all out-of-class samples into outliers; aimed at MPC-style joint training settings where two parties each hold 50% of the data.
   • Loss clipping (Section 4.3.7): As a simple defense, clip each sample's cross-entropy loss to an upper bound C (experiments use C=1 and C=10⁻³) to limit the influence of outliers; the authors state it is computationally more efficient than differentially private gradient clipping.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Validated three types of attacks on CIFAR-10/100, Adult, and WikiText-2; poisoning improves leakage by 1–2 orders of magnitude.

   Experimental Setup

   • Models tested: Wide-ResNet on CIFAR-10/100 (100 epochs, average test accuracy 91%/67%); a three-layer feedforward network on Adult (84% test accuracy); a small GPT-2 trained on WikiText-2; kNN and linear SVM for the theoretical part.
   • Datasets: CIFAR-10/CIFAR-100 (50,000 samples each), Adult (48,842 users), WikiText-2 (about 3 million tokens); the appendix additionally has Texas100 (SVM) and Fashion MNIST (SVM white-box).
   • Baselines: for membership inference, the shadow-model attack of Carlini et al. [11] and the global threshold attack of Yeom et al. [75]; for attribute inference, Mehnaz et al. [45] and an imputation baseline; for canary extraction, Carlini et al. [13].
   • Metrics and protocol: membership/attribute inference report [email protected]% FPR and FPR at fixed TPR=50%; canary extraction reports exposure (bits) and median number of guesses.
   • Sample sizes and repetitions: for CIFAR, 250 target points are randomly selected per experiment, and N=128 models are trained for 128-fold leave-one-out cross-validation; for Adult, 500 target points, 10 target models plus 128 shadow models are selected; for the canary experiments, 45 canaries are injected and 45 target models are trained (2,025 prefix-secret combinations in total).

   Main Results

   Swipe horizontally to read the table

   Attack	Setting	Without poisoning	With poisoning	Source
   Membership inference [email protected]%FPR	CIFAR-10, r=8 (0.03%)	7%	59%	Figure 2
   Membership inference [email protected]%FPR	CIFAR-100, r=16	22%	69%	Main text 4.2
   Membership inference FPR@TPR=50%	CIFAR-10	24%	0.05% (480×)	Main text 4.2
   Membership inference FPR@TPR=50%	CIFAR-100	2.5%	0.025% (100×)	Main text 4.2
   Attribute inference [email protected]%FPR	Adult sex, r=16	1%	30%	Figure 10
   canary median number of guesses	WikiText-2, prefix+suffix	9,018	230 (39×)	Main text 6.3

   • The authors state that label-flipping poisoning is "very effective": a single poisoning (r=1) raises the TPR on CIFAR-10 by 1.75×.
   • When 250 targeted attacks are run simultaneously, total poisoning reaches 4,000 points, but model test accuracy only drops from 92% to 88%, and the MI success rate on non-target points is unchanged; the authors state that repeating the experiment with 50 target points gives similar results.
   • On attribute inference, the authors point out that without poisoning the attack of [45] is worse than the imputation baseline, and the attack first exceeds that baseline after r≥4 poisonings.

   Mechanism Analysis: Poisoning Turns Inliers into Outliers

   • What was measured: Figure 3 plots the loss distributions of 5 CIFAR-10 sample members/non-members as the number of poisonings varies; Figure 4 reruns the attack (r=4) on the 5% of samples that were originally hardest/easiest to infer.
   • Results: Poisoning shifts confidence left and makes the two class distributions easier to separate; for the 5% of samples originally hardest to infer, [email protected]%FPR rises from 0.1% to 43% (430×); for the samples originally easiest to infer (baseline already at 73% TPR), the improvement is small.
   • Authors' interpretation: The attack lowers the privacy of the dataset's average case to the worst-case privacy of outliers, so even inlier users may have their membership inferred.

   Ablation on Shadow Models and Label Choice

   • Shadow models no longer necessary: with poisoning, the global threshold attack of Yeom et al. [75] on CIFAR-10 is close to [11] with 128 shadow models; at FPR=0.1%, the uncalibrated attack without poisoning has TPR 0.1% (random level), and reaches 43% after 16 poisonings (Figure 5).
   • Mislabelling choice: on CIFAR-10/100, "random" (the same random mislabel) is slightly better than "best" (most likely wrong class) and "worst" (least likely wrong class); "random-multi" (a different mislabel for each copy) is worse than the no-poisoning baseline at low FPR (Figures 6, 16). The authors state that the key is consistency of the mislabel.
   • Modifying the target sample: Appendix A.2 tries poisoning that modifies x by adding noise, data augmentation, unadversarial samples, etc. (r=8), all of which are worse than directly mislabelling the original sample; the authors state that they could not find a better strategy and leave it as an open question.

   Defenses, Partial Knowledge and Untargeted Attacks

   • Loss clipping: at C=1, the targeted attack is suppressed at low FPR (r>1 is instead harmful), but at C=10⁻³ poisoning still raises the attack success rate by 3×, and test accuracy drops from 91% to 87% (Figures 7, 8).
   • Partial knowledge: when the attacker only knows the target's nearest-neighbor images (nearest neighbors in CLIP features), the TPR of both the baseline and the r=4 poisoning attack drops by less than 1.6× (Figure 19).
   • untargeted: when the attacker controls 50% of the data and mislabels all of it as the same class, on CIFAR-10 the [email protected]%FPR over all victim points rises from 9% to 16%, and at TPR=50% the FPR drops from 24% to 3% (8×); for the 5% of points hardest to infer, TPR rises from 0.1% to 3.7% (37×); clipping at C=1 is ineffective against the untargeted attack (Figures 9, 25).

   Other Ablations and Analyses

   • canary calibration: 2 shadow models raise exposure from 3.1 to 7.1 bits (guess count drops 16×), and the recovery rate within 100 guesses rises from 0.1% to 10% (Figure 11).
   • Prefix distribution: code, random tokens, and worst-loss prefixes give higher canary exposure, while foreign languages and best-loss prefixes are ineffective (Figure 12).
   • suffix poisoning: a zero-padded suffix inserted 64 times (0.3% of the data) raises exposure from 7.4 to 9.2 bits; the prefix+suffix combination reaches 11.4 bits, and the recovery rate within 100 guesses rises from 10% to 42% (Figure 13).
   • Poisoning improves exposure only when the number of known prefix tokens is ≥8 (Figure 14).
   • Without shadow models, poisoning raises exposure from 3.1 to 7.3 bits, comparable to the gain from shadow models (7.4 bits), and combining the two reaches 11.4 bits (main text 6.4).
   • SVM white-box: 200 poisonings raise the number of support vectors of linear/polynomial SVM from 121/176 to 512/642 (4.2×/3.7×, 5 trials, Appendix D).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list open directions such as defense robustness and stronger poisoning strategies; experiments are limited to several datasets and small models.

   Limitations and future directions noted by the authors

   • The attack does not aim to evade specific poisoning defenses; the authors state that whether the attack remains robust to loss clipping and more complex data sanitization mechanisms is a question for future work (Section 2.3).
   • Poisoning strategies that modify the target sample itself did not improve the attack; the authors leave as an open problem the design of stronger privacy poisoning strategies that change the target sample (Section 4.3.5).
   • Attacks with more constraints (e.g., clean-label poisoning) are left for future work (Section 3.1.1).
   • The authors hope future work will further clarify the interaction between data poisoning and privacy leakage in machine learning (Section 7).
   • The authors point out that average-case canary auditing may underestimate worst-case leakage under small-scale poisoning, and that worst-case auditing methods may more accurately measure the privacy of most users (Section 7).

   Scope of experiments

   • The models tested are Wide-ResNet (CIFAR-10/100), a three-layer feedforward network (Adult), and a small GPT-2 (WikiText-2); theoretical validation uses kNN and linear SVM, 5 categories in total.
   • The datasets are CIFAR-10, CIFAR-100, Adult, and WikiText-2, with Texas100 and Fashion MNIST additionally in the appendix (SVM experiments).
   • Defense evaluation includes only loss clipping (C=1 and C=10⁻³); differential privacy is only discussed and not empirically tested, and the authors state it provably prevents the attack.
   • Membership inference experiments train 128 models per setting for 128-fold cross-validation; Adult uses 10 target models plus 128 shadow models; canary experiments use 45×45=2,025 combinations.
   • The paper does not report the runtime or computational cost of the attack; the untargeted attack is evaluated only in the setting where the attacker holds 50% of the data.

   Collapse this answer

6. 06Summarize the main content of the paper

   Poisoning training data can turn others' samples into outliers, raising leakage from three types of privacy inference by 1–2 orders of magnitude.

   A CCS 2022 attack paper that proposes "active inference attacks": the attacker injects a small amount of poisoning data into the training set, amplifying privacy leakage about other parties' training samples, connecting two previously independent lines of research: data integrity and data privacy.

   • Problem: Can an attacker who can "write" the training set thereby "read" others' private samples. The threat model requires only a one-time injection of arbitrary poisoning points (at minimum 1), after which the attacker only queries the model and does not participate in training.
   • Method: The core is to turn the target sample into an outlier—for image/tabular tasks, insert consistently mislabeled copies of the target; for language models, poison with out-of-distribution prefixes or repeated suffixes; combined with shadow models to calibrate the loss. On kNN and linear SVM, the authors prove 100% membership inference accuracy is achievable.
   • Membership inference: On CIFAR-10, 8 poisonings (0.03% of the training set) raise [email protected]%FPR from 7% to 59%; at fixed TPR=50%, FPR drops from 24% to 0.05% (480×).
   • Property inference: Inferring sex on the Adult dataset, 16 poisonings raise [email protected]%FPR from 1% to 30%, while the no-poisoning baseline is even worse than pure data-dependent imputation.
   • Canary extraction: On WikiText-2 with a small GPT-2, prefix+suffix combined poisoning reduces the median number of guesses for a 6-digit canary from 9,018 to 230 (39×).
   • Conclusions and implications: The authors state the attack pushes average-case privacy toward worst-case, making otherwise safe inliers inferable; poisoning also eliminates the overhead of shadow models. The authors argue this undermines the relevance of the "ideal functionality" privacy guarantee in MPC joint training—a malicious party honestly executing the protocol can amplify leakage merely by choosing its inputs; differential privacy can prevent such attacks, while simple loss clipping only partially mitigates and harms accuracy.

   Collapse this answer

Related papers