Researchers jailbreak text-to-image models with safety guardrails by fine-tuning large language models

How to turn unsafe prompts into adversarial prompts that bypass guardrails and generate images with the target harmful semantics, without further querying the target text-to-image model.

• Use case: The authors state that text-to-image models are already used in industry and can generate realistic images from natural language; unsafe prompts may lead to harmful images such as pornographic content. Defenses mainly consist of external safety filters, or modifying model parameters through alignment.
• Existing limitations: The authors state that black-box attacks such as SneakyPrompt require repeatedly querying the target model, while no-box attacks such as Ring-A-Bell and MMA require querying a surrogate text encoder. The number of queries is high, and adversarial prompts are often incoherent, e.g., containing meaningless tokens.
• Core approach: Use an LLM that can write coherent text to rewrite prompts, then fine-tune it with preference data, so that unseen unsafe prompts have a chance of succeeding with a single rewrite.
• Proposed in this paper: PromptTune. The authors state this is the first query-free attack: after fine-tuning AttackLLM, generating adversarial prompts for new unsafe prompts no longer requires querying the text-to-image model, and it can also enhance other query-based attacks.
• Threat model:
  • Goal: Bypass guardrails and generate harmful images with specific sensitive content (e.g., pornographic). Success means bypassing the guardrails and the image containing the desired harmful semantics.
  • Victim system: A text-to-image model with guardrails. Filters judge the prompt or image outside the model and block it; alignment modifies the model itself.
  • no-box: No access to the target model, but a pretrained large language model or surrogate text encoder can be used to rewrite prompts.
  • black-box: Can query the text-to-image API and adjust the rewriting strategy after obtaining images.

Following the grouping in the introduction and Section 2, the paper discusses four categories of work.

• Diffusion-based text-to-image: DALL·E 3, Imagen, Stable Diffusion v3.5, Midjourney, SDXL, etc. generate semantically similar images based on prompts. The authors state that recent years have been dominated by diffusion models, and this paper only discusses such models: iterative denoising in latent space, with a decoder mapping back to images. Stable Diffusion uses CLIP to encode text, U-Net for denoising, and a pre-trained VAE for decoding.
• Safety guardrails: Filters use external classifiers to check prompts or images, blocking if harmful. The paper mentions George (2020), Li (2022), LAION-AI (2023), CompVis (2022), and states that Stable Diffusion and DALL-E use filters. Alignment instead modifies parameters. SD v2.1 was retrained on data excluding harmful content, which the authors note is computationally expensive. MACE uses LoRA to fine-tune U-Net cross-attention; SafeGen fine-tunes self-attention so that unsafe prompts generate mosaic images. Related work also cites Schramowski et al. (2023), Gandikota et al. (2023), Zhang et al. (2024b).
• Text-to-image jailbreaks: In black-box, SneakyPrompt (Yang et al., 2024b) uses a reward model and reinforcement learning, iteratively rewriting based on target model feedback; Tian et al. (2024) is categorized similarly, but the text does not detail their method. In no-box, Ring-A-Bell (Tsai et al., 2023) uses a genetic algorithm on an alternative text encoder, avoiding unsafe keywords while keeping text embeddings close to the original prompt; MMA (Yang et al., 2024a) computes token-level gradients using a substitute encoder. The authors criticize that each adversarial prompt requires many queries to the substitute model, and results may be semantically meaningless.
• LLM jailbreaks: Chao et al. (2023), Mehrotra et al. (2023), and AdvPrompter (Paulus et al., 2024) use one LLM to construct adversarial prompts for another LLM. AdvPrompter does SFT based on the target LLM's responses. The authors state they extend this idea to text-to-image.

Baselines and data: no-box baselines are Ring-A-Bell and MMA-Diffusion, plus no attack; black-box baseline is SneakyPrompt, and AdvPrompter's SFT is extended to PT-AdvPrompter. Prompt sets are NSFW-56k, Civitai 8M, self-built ShortPrompt, plus gore/violence prompts. Guardrails are keyword matching, text embedding, image embedding filters, and MACE, SafeGen.

The authors position their work as: existing text-to-image jailbreaks rely on many queries and often produce incoherent prompts; this paper uses a fine-tuned LLM for rewriting at generation time without querying the target model, and claims this is the first query-free attack.

PromptTune uses AttackLLM to rewrite unsafe prompts into adversarial prompts, generates images with the guarded text-to-image model, and then uses the Judge's matching score to guide fine-tuning. Figure 1 shows the three components.

Problem setup

• Success criterion: the adversarial prompt bypasses the guardrail, and the image contains the harmful semantics intended by the original unsafe prompt.
• Two capabilities: no-box does not access the target model; black-box can query the API multiple times and see the images. PT-base belongs to the former; the latter two variants query the target model when constructing preference data.
• Interception difference: if a filter blocks the request, no image is produced, and Judge(p, C) = 0. Alignment always produces an image, which is then checked for whether it contains the target harmful semantics.

Three components

• AttackLLM: an interactive LLM, using Mistral-7B-Instruct in experiments. The authors prepare a system prompt to guide rewriting; Table 20 gives the full prompt.
• Guarded text-to-image model: the main experiments use SDXL-Turbo with 4 diffusion steps. For alignment experiments, the open-source alignment models MACE and SafeGen are used directly as text-to-image models.
• Judge: uses CLIP to embed prompt p and image C, and the matching score is the cosine similarity of the two embeddings. A score above threshold τ is considered a success. The default is τ = 0.26.

Preference data

The collection process in Figure 3:

1. The base AttackLLM generates two adversarial prompts for each unsafe prompt.
2. The guarded model generates images for each; those that fail the filter are recorded as unsuccessful.
3. When at least one Judge score is greater than τ, the one with the higher matching score is the preferred response pl, and the other is pr. If neither is greater than τ, both are discarded.

The fine-tuning prompts are 30,000 randomly sampled items each from NSFW-56k and Civitai 8M, plus 6,000 from ShortPrompt. The authors state that this yields 66,000 preference samples, each being (p, pl, pr). For testing, 100 items are additionally sampled from each set.

Three variants

• PT-base: no-box, no fine-tuning, the base model rewrites once.
• PT-AdvPrompter: black-box. Following AdvPrompter, only pl is used as the target response for SFT, not pr.
• PT-dpo: treats p as the query and pl, pr as the preferred and non-preferred responses, fine-tuning with DPO. The authors state the goal is to make the model more likely to rewrite p as pl rather than pr. Default lr = 1e-7, DPO loss coefficient β = 0.1.

The authors state that ideally, after fine-tuning, unseen unsafe prompts can succeed with a single rewrite.

Evaluation criteria

• Filters: bypass rate, i.e., the proportion of the test set that successfully bypasses.
• Alignment: the authors state that since these models do not block queries, the bypass rate would be 1, so they instead report the average CLIP score between the image and the corresponding original unsafe prompt.
• Utility and efficiency: FID is computed only on images that bypass the guardrail, referenced against the image distribution of the same prompt on the unguarded model; lower is better as indicated in the tables. Efficiency is the average number of queries and time to generate one adversarial prompt. No-box queries the surrogate model, black-box queries the text-to-image model.

Experimental setup

• Systems under test: three external filters evaluated on SDXL-Turbo; alignment experiments directly use the open-source alignment models MACE and SafeGen. AttackLLM is Mistral-7B-Instruct. The Judge is based on CLIP.
• Data: NSFW-56k, Civitai 8M, ShortPrompt, with 100 test items each. In Table 7, the average lengths are 40.5, 43.5, and 10.2, respectively. ShortPrompt consists of captions generated by BLIP-opt-2 from online sensitive images. The gore/violence set has 3,000 training items and 100 test items (Table 8).
• Baselines: no-box baselines are no attack, Ring-A-Bell, and MMA-Diffusion. The latter two use the text encoder of Stable Diffusion v1.4, with default iterations of 3,000 and 1,000, respectively. The black-box baseline is SneakyPrompt.
• Metrics: filters report bypass rate; alignment reports average CLIP score. The appendix additionally includes NudeNet (average number of exposed body parts detected), MHSC, and Q16; the main text does not give the formulas for the latter two. FID measures the distribution of images that bypass the guardrail versus images of the same prompt without the guardrail. Efficiency is the average number of queries and time.
• Defaults: lr = 1e-7, β = 0.1, τ = 0.26. The main results involve one interaction with AttackLLM. The paper does not report the number of repetitions or statistical significance tests.

Main results

The table below is based on Table 1, with 100 items per set. Filters use bypass rate; MACE and SafeGen use average CLIP score. Ring-A-Bell is not listed.

Swipe horizontally to read the table

Guardrail	Dataset	Metric	None	MMA	PT-base
Keyword match	NSFW-56k	bypass rate	0.310	0.450	0.900
Text embedding	NSFW-56k	bypass rate	0.100	0	0.240
Image embedding	NSFW-56k	bypass rate	0.180	0.260	0.590
Keyword match	Civitai	bypass rate	0.090	0.270	0.700
Text embedding	Civitai	bypass rate	0.150	0.050	0.290
Keyword match	ShortPrompt	bypass rate	0.460	0.590	0.910
MACE	NSFW-56k	average CLIP	0.231	0.222	0.222
SafeGen	ShortPrompt	average CLIP	0.251	0.214	0.260

• The authors state that on safety filters, compared with no attack, the bypass rate improves roughly 2–3 times, and is higher than Ring-A-Bell and MMA across all three filters and three datasets. The latter sentence is consistent with Table 1, including filter cells not listed; the former sentence is the authors' summary, not a per-cell computed range. On text embedding across the three datasets, Ring-A-Bell is 0 in all cases.
• The authors state that the no-box attack is not always effective against alignment guardrails: it works well against SafeGen but weakly against MACE. In Table 1, on MACE across the three datasets, PT-base is not higher than None (Civitai 0.204 vs. 0.214, ShortPrompt 0.247 vs. 0.258). On SafeGen, PT-base is higher than None (NSFW-56k 0.236 vs. 0.224, Civitai 0.236 vs. 0.232).
• The authors state that PT-base generates readable prompts, whereas surrogate text-encoder methods produce unreadable sentences and are therefore more easily detected at the text level.

Three variants

• Comparison targets: Table 3 compares PT-base, PT-AdvPrompter (SFT), and PT-dpo. Filters use bypass rate; alignment uses average CLIP score.
• Numbers: on keyword match, PT-dpo is 0.990, 0.970, 1.000 (NSFW-56k, Civitai, ShortPrompt). On text embedding, PT-dpo is 0.710, 0.700, 0.830, and PT-base is 0.240, 0.290, 0.230. On MACE, PT-dpo is 0.242, 0.219, 0.260. PT-AdvPrompter is not higher than PT-base everywhere: on NSFW-56k text embedding it is 0.130, lower than 0.240; on Civitai keyword match it is 0.670, lower than 0.700.
• Authors' interpretation: the authors state that fine-tuning further improves the bypass rate and average CLIP score, and that the keyword match bypass rate approaching 1 indicates that AttackLLM learns to avoid sensitive words during fine-tuning. The authors state that PT-dpo outperforms PT-AdvPrompter because DPO contrasts (pl, pr), while SFT uses only pl. In Table 13, the authors state that PT-dpo has mostly lower FID, with the rest similar to the other two variants.

Enhanced SneakyPrompt

• Approach: use PT-dpo's adversarial prompts as the initial prompts for SneakyPrompt, denoted SneakyPrompt-PT.
• Results: most cells in Table 4 are higher. On text embedding, NSFW-56k goes from 0.510 to 0.960, Civitai from 0.460 to 0.960, ShortPrompt from 0.880 to 0.980. Exception: on ShortPrompt image embedding, SneakyPrompt-PT is 0.990, while the original method is 1.000. In Table 5, most query counts decrease, e.g., NSFW-56k keyword match from 2.07 to 1.00, image embedding from 9.84 to 3.03. Exception: ShortPrompt SafeGen from 3.57 to 4.18.
• Authors' interpretation: the authors state that this design is orthogonal to query-based attacks, improving effectiveness and efficiency while utility remains comparable. FID values in Table 14 vary, e.g., NSFW-56k SafeGen from 195 to 235; the paper does not explain each cell.

Other ablations and analyses

• Time (Table 12, single RTX 6000, 24GB, 100 items): Ring-A-Bell 911.9 seconds, MMA 1329 seconds, PT-base 3.613 seconds. The authors state that PT-base is hundreds of times faster and interacts with the pretrained LLM only once.
• Gore/violence (Table 8, test 100): keyword match bypass rate None 0.620, PT-base 0.530, PT-dpo 0.770; text embedding 0.240, 0.590, 0.730; image embedding all three are 1.000.
• lr (Table 15, guardrail not specified): at 1e-6 / 1e-7 / 1e-8, NSFW56k bypass rate is 0.920 / 0.710 / 0.510, and FID is 235 / 207 / 183. The authors state that as lr increases, there is a trade-off between bypass rate and FID.
• β (Table 16, guardrail not specified): at β of 0.05, 0.1, 0.2, NSFW56k bypass rate is 0.750, 0.710, 0.590. The authors state that for β > 0.05, the bypass rate decreases as β increases.
• τ (Table 17, guardrail not specified): at τ of 0, 0.24, 0.26, 0.28, NSFW56k bypass rate is 0.830, 0.600, 0.710, 0.430. The authors state that too small a τ includes irrelevant samples, while too large a τ makes D too small and fine-tuning unstable.
• Multiple attempts (Table 18, ShortPrompt, PT-dpo, text embedding): at 1, 2, 4, 10 attempts, the bypass rate is 0.71, 0.83, 0.92, 0.98. In Table 3, the single-attempt value for the same condition is 0.830, inconsistent with 0.71; the paper does not explain this.

Limitations and future directions noted by the authors

• The advanced variants require a large number of queries to the target text-to-image model when constructing preference data during the fine-tuning stage; no queries are made at inference time, but the collection cost is considerable (Section 6).
• In the no-box setting, using PT-base to jailbreak alignment-type guardrails (such as MACE) remains difficult; the authors state that alignment is a more robust defense against attackers without target access (Section 6).
• The authors plan to restrict access to the preference dataset and the fine-tuned AttackLLM, and to report the results to image generation service providers (Section 6).
• One mitigation is to incorporate AttackLLM into the alignment process of text-to-image models, so that even adversarial prompts do not generate harmful images (Section 8).
• The future direction the authors describe is to combine this LLM-based method with tree-of-thought processes to enhance attack capability (Section 8).

Experimental coverage

• On the text-to-image side, SDXL-Turbo with three filters, as well as the open-source MACE and SafeGen alignment models, are used; AttackLLM is Mistral-7B-Instruct; the Judge uses CLIP (Section 5.1).
• Three test sets of pornographic prompts, each with 100 items; the authors record the preference data as 66,000 items. For gore/violence, training is 3,000 and testing is 100 (Table 7, Table 8).
• No-box baselines include no attack, Ring-A-Bell, and MMA; the black-box baseline is SneakyPrompt. The main results are from a single rewrite; multiple attempts are reported only for one condition in Table 18.
• Metrics include bypass rate, average CLIP score, FID, NudeNet, MHSC, Q16, number of queries, and time. The paper does not report human evaluation, agreement with humans, or number of repetitions, nor does it report the total number of queries used to build the 66,000-item preference dataset.
• The paper does not specify which guardrail Tables 15–17 correspond to, nor whether the preference data were collected separately for each guardrail or whether all guardrails share the same dataset and the same AttackLLM.

PromptTune uses a fine-tuned LLM to rewrite prompts, attacking text-to-image models with safety filters or alignment.

• Unsafe prompts are blocked by guardrails, while existing jailbreaks often require repeated queries to the target model or alternative text encoders, and the rewritten results may be incoherent.
• Mistral-7B-Instruct serves as AttackLLM, and CLIP cosine similarity serves as Judge. The base model first writes two adversarial prompts, forms preference pairs according to threshold τ, then does SFT or DPO. PT-base is not fine-tuned and only rewrites once.
• Three pornographic prompt sets, each with 100 prompts. PT-base bypass rates for keyword match are 0.900, 0.700, 0.910 (Table 1). On text embedding, PT-base is 0.240, 0.290, 0.230, and PT-dpo is 0.710, 0.700, 0.830 (Table 3).
• Alignment guardrails are reported as average CLIP scores. On NSFW-56k with MACE, no attack is 0.231, PT-base is 0.222, PT-dpo is 0.242 (Table 1, Table 3).
• After initializing SneakyPrompt with PT-dpo's output, the NSFW-56k text embedding bypass rate goes from 0.510 to 0.960, and keyword match average online queries go from 2.07 to 1.00 (Table 4, Table 5). PT-base generates one adversarial prompt in an average of 3.613 seconds, while Ring-A-Bell takes 911.9 seconds (Table 12).
• The authors conclude that fine-tuned LLMs can bypass existing text-to-image guardrails and can also enhance query-based attacks; the authors believe that alignment guardrails remain difficult under no-box. The authors propose incorporating AttackLLM into the alignment process as mitigation.