Extracting Training Data from Diffusion Models

1. 01What problem does this paper attempt to solve?

   Examine whether diffusion models memorize and reproduce training images, and how the degree of memorization changes with modeling and data decisions.

   Core question: whether image diffusion models memorize individual images in the training set and reproduce them during generation, thereby undermining privacy assumptions.

   • Scenario and importance: The authors note that prior large-scale training "did not find overfitting to be a problem" (citing Saharia et al.), and that some research suggests using diffusion models to generate synthetic data to "protect the privacy of real images"; if models memorize training data, all these privacy guarantees fail, and copyright and "digital forgery" issues are also involved.
   • Shortcomings of existing methods: Training data extraction research has mainly focused on text language models, and verbatim definitions are not suitable for high-resolution images; privacy research on image generative models mostly focuses on GAN generalization or semantic similarity ("style copying"), lacking a characterization of worst-case privacy for diffusion models.
   • Core observation: The authors hypothesize that diffusion models memorize training images, and that images repeated many times are memorized with orders of magnitude higher probability, so biasing the search toward repeated samples can greatly reduce attack cost.
   • Contributions of the paper: Propose memorization definitions applicable to images and a two-stage data extraction attack, extracting training images on Stable Diffusion and Imagen; and train hundreds of CIFAR-10 diffusion models to analyze the impact of model accuracy, hyperparameters, data augmentation, and deduplication on privacy, and compare with GANs.
   • Threat model (Section 3.1):
     • Goals: From strong to weak: data extraction (recovering an image nearly identical to a training image), data reconstruction (recovering the full image given partial knowledge of the image), membership inference (determining whether an image is in the training set).
     • Knowledge: When attacking conditional generators, assume the attacker knows the captions of some training images, to study worst-case privacy.
     • Capabilities: Black-box attackers can query Gen to generate images and provide arbitrary prompts, but cannot control the internal randomness r; white-box attackers have full access to Gen and fθ, can control randomness, and can denoise arbitrary inputs.
   • Victim systems: Unconditional or conditional (class label/text) diffusion model generation systems.

   Collapse this answer

2. 02What related research exists?

   Related work covers language model memorization extraction, GAN memorization and generalization, training data privacy attacks, and parallel diffusion model privacy research.

   Memorization extraction in language models

   • Carlini et al. [11], Lee et al. [47], Kandpal et al. [41, 40]—extract training text using two-step attacks on language models, with a method pipeline similar to this paper; the paper notes that this paper's difference is that it targets the image domain and uses more semantic regeneration criteria such as CLIP scores rather than verbatim repetition.
   • Carlini et al. [10]—evaluate language model memorization using canary insertion, an idea borrowed in Section 7.3 of this paper for diffusion model auditing.

   Memorization and generalization in image generation

   • GAN generalization and similarity metric direction—Heusel et al. [32], Arora et al. [3], etc. propose metrics of similarity to the training set, and other work studies mode collapse [61, 15], the influence of individual training samples [4, 75], and when GANs reproduce training samples and how to mitigate it [50, 26]; the paper states that this paper extends these research questions to conditional diffusion models.
   • Tinsley et al. [70]—show that StyleGAN can generate faces of real individuals; Somepalli et al. [65]—show that Stable Diffusion outputs images semantically similar to the training set. The paper notes that this paper identifies privacy issues on more systems (Imagen, CIFAR models) and more threat models (e.g., membership inference), and adopts a stricter definition of memorization.

   Training data privacy attacks

   • Membership inference [62, 80, 8], model inversion [27, 81], attribute inference [28], data extraction [10, 11, 5]—this paper applies all three types of attacks to diffusion models; membership inference directly uses the loss threshold of Yeom et al. [80] and the LiRA of Carlini et al. [8].
   • Feldman [24]—proposes the theory that memorization in classifiers is necessary for generalization; the authors state that this paper's results suggest this theory may extend to generative models.

   Parallel diffusion model privacy research

   • Wu et al. [78], Hu et al. [34]—perform membership inference on diffusion models; the paper states that this paper uses more sophisticated attack methods and studies data extraction, a stronger privacy risk.
   • Somepalli et al. [65]—mainly compares semantic similarity between generated images and the training set ("style copying"); the paper states that this paper focuses on a stricter no- (original text: "a much more restrictive notion of memorization") worst-case privacy and covers more models.

   Baselines and benchmarks: comparison baselines are GAN models (five pretrained models StyleGAN-ADA, DiffBigGAN, E2GAN, NDA, WGAN-ALP, plus self-trained StyleGAN, MHGAN, BigGAN) and two pretrained DDPMs (OpenAI-DDPM, DDPM); the experimental benchmark is CIFAR-10, and large model experiments use Stable Diffusion and Imagen trained on LAION. The authors position this paper as: extending language-model-style extraction attacks and GAN-style generalization analysis to diffusion models, and providing a privacy comparison across model forms (diffusion vs GAN).

   Collapse this answer

3. 03How does the paper address this problem?

   Define image memorization, extract training images using a two-stage "generate-and-filter" attack, and then decompose the factors influencing memorization through controlled experiments.

   Overall approach: first give a definition of memorization applicable to high-resolution images, then use a two-stage generate-and-filter pipeline to extract training images from Stable Diffusion and Imagen, and finally train hundreds of models on CIFAR-10 for membership inference, inpainting reconstruction, GAN comparison, and defense evaluation.

   Memorization definition

   • (l, δ)-Diffusion Extraction (Definition 1, adapted from [11]): x is said to be extractable from the model if there exists an efficient algorithm A that does not receive x as input such that x̂ = A(fθ) and l(x, x̂) ≤ δ. The default distance is the normalized Euclidean 2-norm l₂(a,b) = √(∑_i (a_i − b_i)² / d), where d is the input dimension, normalizing the distance to [0,1].
   • (k, l, δ)-Eidetic Memorization (Definition 2): x is extractable, and at most k neighbors in the training set satisfy l(x, x̂) ≤ δ; k measures the number of near-duplicates in the training set.
   • The authors emphasize that the definition is deliberately conservative: for example, being able to generate a recognizable new photo of Obama but not a near-copy of a specific training image does not count as memorization (Figure 2).

   Large model extraction pipeline (Stable Diffusion / Imagen)

   1. Locate duplicate samples: use CLIP to embed training images into 512-dimensional vectors for all-pairs comparison (the authors claim an efficiency improvement of over 1500×), determine near-duplicates by embedding cosine similarity, and take the captions of duplicate images as attack prompts.
   2. Mass generation: black-box query Gen, generating 500 candidates per prompt; Stable Diffusion uses default PLMS sampling at 512×512 resolution, and uses a timestep-resampled generation implementation (the authors claim about 10× speedup with slightly reduced visual fidelity).
   3. Membership inference filtering: build a graph over the 500 generations for the same prompt, connecting two images if they are close under distance d, and judge as a memorized image when the maximum clique is at least 10 images; distance d uses a blockwise 2-norm—dividing the image into 16 128×128 tiles and taking the maximum of the 2-norm distances between corresponding tiles of the two images, to reduce false similarity caused by gray backgrounds etc.

   CIFAR-10 controlled experiments

   • Experimental subjects: using the OpenAI Improved Diffusion code or its reimplementation, train 16 diffusion models on a random half of the CIFAR-10 training set; class-conditional models have FID below 3.5, unconditional models have FID below 4.
   • Untargeted extraction: generate 2^16 images per model, 2^20 candidates in total; because the global 2-norm threshold fails (Figure 6), use an adaptive threshold l(x̂, x; S_x̂) = (l₂(x̂, x))/(α · E_{y ∈ Sx̂}[l₂(x̂, y)]), i.e., the distance from the generated image to the nearest training image divided by the average distance from that generated image to its n nearest training neighbors, with α = 0.5, n = 50.
   • Membership inference: implement loss threshold [80] and LiRA [8]; the loss is the diffusion reconstruction loss L(x, t, ε), with t = 100 selected after scanning over 16 shadow models; the improved version (Strong LiRA) performs Monte Carlo averaging over up to 20 noise samples and horizontal flip augmentation on images.
   • Inpainting reconstruction attack: mask the left half of the image, generate n = 5,000 reconstructions using the inpainting algorithm of Lugmayr et al. [48], score by the ratio of the main model's loss (at t = 100) to the loss of a shadow model that has not seen the image, and take the top-10.

   Comparison and defense evaluation

   • GAN comparison: for GANs, use the discriminator loss for loss threshold and LiRA (training 256 models, each on a random half of CIFAR-10); extraction experiments use 5 pretrained GANs and 2 pretrained DDPMs, generating 2^20 images each, counted with the same similarity function as for diffusion models.
   • Defenses: deduplicate CIFAR-10 with imagededup at similarity > 0.85 (removing 5,275/50,000 images) and retrain; try DP-SGD; for canary auditing, insert uniform noise images, insert them with different repetition counts, and measure maximum exposure.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Extract training images from Stable Diffusion/Imagen, and conduct membership inference, reconstruction, GAN comparison, and defense evaluation on CIFAR-10.

   Experimental setup

   • Models tested: Stable Diffusion v1.4 (890 million parameters, trained on 160 million images), Imagen (2 billion parameters); 16 class-conditional and unconditional diffusion models self-trained on CIFAR-10; pretrained GANs (StyleGAN-ADA, DiffBigGAN, E2GAN, NDA, WGAN-ALP) and pretrained DDPMs (OpenAI-DDPM, DDPM); self-trained StyleGAN, MHGAN, BigGAN (256 each).
   • Datasets: LAION (Stable Diffusion training set, 160 million images), Imagen training set (not public), CIFAR-10 (50,000 images).
   • Metrics: (l2, δ)-extraction (δ = 0.15 for Stable Diffusion, δ = 0.1 for eidetic analysis), membership inference TPR@FPR and AUC, FID, canary exposure.
   • Review method: Stable Diffusion results are manually annotated—each ranked generated image is compared against the training set to determine "extracted / not extracted", and the top-1000 generations are additionally manually visually inspected; the paper does not report the number of annotators or agreement.
   • Sample sizes: For Stable Diffusion, 350,000 of the most repeated prompts each generate 500 images (175 million total); for Imagen, the top-1000 repeated prompts each generate 500 images; for CIFAR-10, 2^16 images per model, about 1 million total; the inpainting attack is evaluated on 100 images.
   • Number of repetitions: CIFAR-10 membership inference uses 16 models; the paper does not report the number of repetitions for the Stable Diffusion/Imagen extraction experiments.

   Main results

   Swipe horizontally to read the table

   Model	Generations	Extracted images	FID	Source
   StyleGAN-ADA	1M	150	2.9	Table 1
   DiffBigGAN	1M	57	4.6	Table 1
   E2GAN	1M	95	11.3	Table 1
   NDA	1M	70	12.6	Table 1
   WGAN-ALP	1M	49	13.0	Table 1
   OpenAI-DDPM	1M	301	2.9	Table 1
   DDPM	1M	232	3.2	Table 1

   • The authors state that diffusion models memorize more than GANs: OpenAI-DDPM and StyleGAN-ADA have the same FID of 2.9, but the extraction counts are 301 versus 150 (about 2 times); and generative models with better (lower) FID memorize more.
   • Stable Diffusion: Of 175 million generations, 94 satisfy (l2, 0.15)-extraction, and manual inspection adds 13 more, for a total of 109; the attack can identify 50 memorized images at 0 false positives, and all memorized images can be identified when precision is above 50% (Figure 4).
   • Imagen: 23 of the top-1000 repeated prompts are judged memorized; the authors state that this proportion is higher than Stable Diffusion, and believe it is related to larger model capacity, longer training, and a smaller dataset.

   Membership inference and the utility-privacy relationship

   • What was tested: Loss threshold and LiRA on CIFAR-10 class-conditional diffusion models, and Strong LiRA with Monte Carlo averaging and horizontal flipping added (Figure 10).
   • Results: Original LiRA exceeds 70% TPR at FPR 1% (AUC 0.982), loss threshold AUC 0.613; Strong LiRA raises TPR from 7% to 44% at FPR 0.1% (AUC 0.997). The authors note that, for reference, SOTA classifiers have TPR below 20% at FPR 1% (citing [8]).
   • Utility relationship: As FID decreases from about 8 to about 4, TPR@FPR=1% rises from 7% to close to 100% (Figure 11); Appendix Figure 18 shows that as the number of training-processed samples increases from 5M to 102M, TPR@FPR=1% rises from 5% to 99%. The authors state that better diffusion models leak more privacy.

   Inpainting reconstruction attack

   • What was tested: Mask the left half of CIFAR-10 images, generate 5,000 inpainting reconstructions for each of 100 images, and compare the 2-norm distance of reconstructions when the target is in versus not in the training set (Figures 12, 13).
   • Results: When the target is in the training set, the 2-norm distance of the top-10 reconstructions is lower than when it is not in the training set; in Figure 12, for the two examples of bird and cat, the distance is about 0.05–0.10 when the target is in the training set and about 0.15–0.30 when it is not (scatter range in the figure).
   • Authors' interpretation: An attacker with partial knowledge of the image can improve extraction; Appendix D states that the contrast loss (main model loss divided by support model loss) correlates more strongly with 2-norm distance, and that the attack fails when the target is not in the training set, from which the authors conclude that the attack exploits memorization rather than generalization.

   Defense evaluation

   • Deduplication: After deduplicating CIFAR-10 with similarity > 0.85 (removing 5,275 images) and retraining, the extraction count from 1 million generations drops from 1,280 to 986; the authors state that deduplication can mitigate but is insufficient, and expect it to be more effective on larger datasets.
   • DP-SGD: The authors report that applying DP-SGD on CIFAR-10 causes training to consistently diverge, even with a privacy budget ε as high as about 50; gradient clipping or noise addition alone also causes training failure.
   • Canary auditing: Insert uniform noise canaries; the maximum exposure is 10, and some canaries reach maximum exposure after being inserted only 2 times (Figure 16); exposure does not strictly increase with the number of repetitions, and the authors believe that random noise canaries may not be the most effective audit samples.

   Other ablations and analyses

   • Loss timestep: Membership inference works best at t ∈ [50, 300], and subsequent experiments fix t = 100 (Figure 9).
   • Number of Monte Carlo samples: As n increases from 1 to 20, TPR@FPR=0.1% rises from 0.071 to 0.294 (Figure 19a).
   • Horizontal flip augmentation: With n = 20 plus augmentation, TPR@FPR=0.1% reaches 0.437 (Figure 19b).
   • Conditional vs unconditional: Conditional models have TPR@FPR=1% of 0.714, unconditional 0.549 (Figure 21); the authors state that it is difficult to determine whether this is due to conditionality itself or to differences in parameter count.
   • Number of repetitions and extraction: On Stable Diffusion, there is almost no eidetic memorization when k < 100, and extraction increases after k ≥ 100 (Figure 5); the authors note that this is an upper bound because the method specifically searches for repeated images.
   • Cross-model joint memorization: Of the 1,280 images memorized by the self-trained diffusion model and the 361 memorized by the self-trained StyleGAN, 244 overlap, versus about 10 expected under random uniform conditions (p < 10^-261); the authors state that some images are inherently less private, and the reason is left for future research.
   • Negative results: Using an outlier method on Stable Diffusion (average distance to 1,000 nearest neighbors in CLIP space) to attempt extraction of the 10,000 most outlier samples found no memorization; on Imagen, 3 of the 500 samples with the highest out-of-distribution scores were memorized.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list open problems in directions such as definitions, DP training, and auditing; experiments cover two large models and CIFAR-10-scale models.

   Limitations and future directions noted by the authors

   • Memorization definition is conservative (Section 4.1): The definition only covers near-copy extraction; cases such as the model generating a new photo of a recognizable person are not counted, and the authors state that a more comprehensive analysis is needed to capture copy definitions closer to human perception (Section 9).
   • Cause of DP-SGD failure to be investigated (Section 7.2): DP-SGD training on CIFAR-10 consistently diverges; the authors explicitly leave further investigation of these failures as future work, and believe that new advances in privacy-preserving training techniques may be needed.
   • Canary design to be improved (Section 7.3): Randomly generated noise canaries may not be the most effective samples for testing diffusion model memorization.
   • Joint memorization phenomenon to be explained (Section 6): Diffusion models and GANs jointly memorize the same set of images (244 overlapping), and the authors state that understanding this phenomenon is a direction for future work.
   • Generalization question remains open (Section 9): Whether large models generate new content or copy/interpolate training data—the authors state that the success of extraction attacks leaves this question open, and hope future work explores how diffusion models copy training data.
   • Deduplication is insufficient (Section 7.1): After deduplication, the extraction count only drops from 1,280 to 986, and the authors state that deduplication is not a perfect solution.

   Experimental coverage

   • The large models tested are two text-conditional models, Stable Diffusion v1.4 and Imagen; controlled experiments are conducted on CIFAR-10 with 16 diffusion models, and the GAN comparison covers 5 pretrained GANs, 2 pretrained DDPMs, and 3 self-trained GAN architectures.
   • The attacks cover three types of targets: data extraction (Stable Diffusion, Imagen, CIFAR-10, GANs), membership inference (CIFAR-10 diffusion models and GANs), and inpainting reconstruction (CIFAR-10, 100 images).
   • Defense evaluation includes three items—data deduplication, DP-SGD, and canary auditing—all conducted at CIFAR-10 scale.
   • Extraction success is determined by manual annotation (the Stable Diffusion part includes manual visual inspection of the top-1000); the paper does not report the number of annotators, inter-annotator agreement, or the number of repetitions for the large-model extraction experiments.
   • The Imagen training set is not public, and the authors note that they cannot show visualization examples of its successful reconstructions (Section 4.3 footnote).

   Collapse this answer

6. 06Summarize the main content of the paper

   Diffusion models memorize and reproduce training images; the authors state that they leak more than GANs, and that existing defenses are insufficient to address this.

   An attack paper on the privacy of training data for image diffusion models, answering the question "do diffusion models memorize and reproduce training images".

   • Problem: Diffusion models are assumed not to memorize training data, and are therefore expected to be used for privacy-sensitive synthetic data generation; the authors test this assumption and analyze how memorization changes with model and data decisions.
   • Method: Propose two image memorization definitions, (l, δ)-extraction and (k, l, δ)-Eidetic memorization; use a generate-and-filter two-stage attack—sample extensively according to known captions, then filter out memorized images using clique-based membership inference; and train hundreds of models on CIFAR-10 for controlled analysis.
   • Large-model extraction: Extract 109 memorized images from Stable Diffusion (175 million generated, including manual review), and 23 from Imagen's top-1000 repeated prompts; the extracted content includes recognizable photos of people (58%), product images (17%), logos/posters (14%), and 35% carry an explicit non-permissive copyright notice.
   • Controlled experiments: Extract 1,280 unique training images from 1 million CIFAR-10 diffusion model generations (2.5% of the dataset); Strong LiRA membership inference achieves TPR of 44% at FPR 0.1%; as FID improves, TPR@FPR=1% rises from 7% to close to 100%.
   • Cross-model comparison: At the same FID (2.9), OpenAI-DDPM extracts 301 images and StyleGAN-ADA extracts 150; the authors state that diffusion models memorize about 2 times as much as GANs, and that models with better generation quality memorize more.
   • Defenses and conclusion: Deduplication only reduces the CIFAR-10 extraction count from 1,280 to 986, and DP-SGD training on CIFAR-10 diverges; the authors argue that existing privacy techniques do not provide an acceptable privacy-utility trade-off, that synthetic data does not bring privacy for free, and recommend deduplication, reducing overtraining, and using attacks for auditing.

   Collapse this answer

Related papers