No Safe Dose: How Training Data Drives Unsafe Image Generation
No Safe Dose: How Training Data Drives Unsafe Image Generation
Quick read
Based on the paper's full text (AI-generated); refer to the original
Researchers from TU Darmstadt, hessian.AI, and other institutions found through controlled experiments that the unsafe output rate of text-to-image models rises monotonically with the proportion of unsafe images in the training data, and that what matters is the proportion rather than the absolute number. The experiments trained the same PRX-1.2B model on datasets with contamination rates of 0%–9.6% and sizes from 100K to 8M, evaluated by four independent safety classifiers; the unsafe rate rose from 16.6% at 0% contamination to 25.5% at 5% contamination. This effect appears only under adversarial prompts; under safe prompts, all models are about 1%. After complete filtering, a 16.6% baseline still remains, mainly attributed to the frozen text encoder; switching to SafeCLIP reduces it to 9.6%, and combined with data filtering this is a 53% relative reduction versus the unmitigated baseline; filtering incurs no quality cost on FID, CLIPscore, or ImageReward. The authors also found that subsequent SFT raises the unsafe rate under each condition by a further 4.0–9.6 percentage points, and they publicly released the trained models, generated images, and safety annotations.
- Unsafe content is inevitably mixed into the training data of text-to-image models, but whether and how the composition of training data directly determines output safety was previously unclear, because proprietary datasets and different training settings confound causal judgment. The authors point out that newer models generate unsafe content at a higher rate, so the priority of data curation relative to other interventions needs to be clarified.
- Fixing the architecture, hyperparameters and evaluation protocol of the same T2I model (PRX-1.2B), only the proportion of unsafe images in the training set (0% to 9.6%) and the dataset size (100K to 8M) were varied, for a total of 7 conditions; generated images were evaluated with 4 independent safety classifiers, and text encoder ablations (T5-Gemma, CLIP, SafeCLIP) and a factorial design were performed to distinguish proportion from absolute count.
- The output unsafe rate rises monotonically with the training contamination proportion, from 16.6% at 0% contamination to 25.5% at 5%, showing sublinear saturation; the operative variable is the proportion rather than the absolute count, so at scales above about 1M the filtering effect is independent of corpus size. This effect appears almost only under adversarial prompts; under safe prompts all conditions are about 1%. Zero contamination still has a 16.6% floor, which drops to 9.6% when SafeCLIP is used, and the dose effect persists across encoders; safety filtering shows no obvious quality loss on FID, CLIPscore, or ImageReward.
- The authors acknowledge that "unsafe" is an operational definition (based on the LlavaGuard nine-category taxonomy) and is not equivalent to legally illegal content; 90% of the prompt test set consists of adversarial prompts and does not reflect the full picture of natural use. Architectural variants, training stages such as RL, other taxonomies, and the protective effect of data filtering against adversarial attacks and malicious fine-tuning are not covered and are left for future research.
Controlled experiments establish a causal link between training data contamination and unsafe text-to-image outputs, and quantify the complementary effects of two types of interventions: data filtering and a safe text encoder.