RedDiffuser: Using reinforcement diffusion to generate visual inputs for auditing VLM multimodal safety failures

What is audited: whether VLM safety alignment remains effective when partially toxic text is paired with visual context.

• Scenario: The authors state VLMs are being deployed in open applications, where inputs are often incomplete, ambiguous, and context-dependent. Visual input can noticeably steer behavior, so text-only auditing is insufficient; in social, user-generated content, and multi-turn interactions, inputs are rarely well-formed and may carry implicit safety risks.
• Existing gap: The authors state existing evaluations are instruction-centric, mainly examining whether explicit jailbreak prompts or adversarial perturbations trigger harmful outputs, with less attention to failures caused by contextual conditions, potentially overestimating safety.
• Core setup: Whether the model can remain safe when partially harmful text is paired with visual context. Unsafe continuations (profanity, insults, or threats) are treated as an observable indicator of contextual-condition failure. The authors state explicitly that this setup is not treated as a conventional jailbreak driven by explicit malicious intent.
• Proposed: RedDiffuser (RedDiff), a black-box reinforcement learning framework that uses diffusion models to generate semantically coherent visual inputs for safety stress testing, extended to auditing when external NSFW guardrails are present.
• Threat model:
  • Goal: To see whether visual context interacts with and amplifies harmful signals in the text, causing unsafe behavior in open-ended continuation.
  • Knowledge: The authors state it is black-box and query-only, corresponding to real API deployments.
  • Capability: The text context y is fixed; the image x is changed. Target-model feedback is used to find semantically coherent multimodal inputs that expose high-risk behavior, and reinforcement learning optimizes the diffusion model.
  • Victim system: The queried VLM. The extended setup also audits whether upstream safety filtering can block harmful multimodal contexts.

The introduction and Related Work divide the discussion into VLM architecture, and existing safety evaluations and jailbreaks.

VLM and Visual Inputs

• LLaVA (Liu et al., 2024): CLIP encoder connected to LLaMA, using GPT-4 synthesized multimodal instruction data for end-to-end fine-tuning.
• MiniGPT-4 (Zhu et al., 2023): Linear projection aligns frozen ViT and Vicuna, first pre-trained on large-scale image-text pairs, then fine-tuned with high-quality data.
• InstructBLIP (Dai et al., 2023): Based on BLIP-2, adds instruction-aware Query Transformer and reorganizes 26 public datasets. The authors state that adding vision broadens the attack surface compared to text-only LLMs; subsequent work cited in the paper shows that visual perturbations and cross-modal inconsistencies can be used to bypass safety mechanisms or elicit harmful behavior.

Jailbreaks and Safety Evaluations

• White-box attacks (grouped into one set of citations by the paper): use gradient information to optimize adversarial inputs.
• Black-box methods (grouped into another set of citations by the paper): rely on prompt engineering, visual role-playing, or typographic manipulation to bypass safety mechanisms. The introduction also lists explicit jailbreak prompts and adversarial perturbations as the main objects of existing evaluations.
• The authors' criticism is attached to these evaluations, not rewriting the cited authors' conclusions: the authors state that these studies point out limitations of current VLM alignment, but evaluations are still mainly framed around explicit attacks, checking whether direct jailbreak prompts or adversarial inputs can trigger harmful outputs, with less attention to harmful multimodal contexts that do not take the form of direct attacks but may still affect generation.

Baseline Methods and Benchmarks

• Baselines: Text-Only (text only), Single-Prompt (Gemini generates one image prompt per text prefix), Greedy-Prompt (then greedy search). Guardrail experiments and CLIPScore comparisons use un-fine-tuned Stable Diffusion.
• Benchmark: The challenge subset of RealToxicityPrompts, 1,199 text prompts related to toxic continuation behavior, treated as fixed text contexts in the audit.

The authors state that this paper does not propose another jailbreak method, but uses RedDiffuser as a diagnostic framework to audit whether alignment remains effective under exposure to harmful contexts.

RedDiffuser does not frame the task as rewriting explicit malicious instructions. After fixing the text context, it first searches for high-risk image prompts, then fine-tunes the diffusion model with reinforcement learning so that the generated images are more likely to elicit unsafe continuations while remaining semantically coherent.

Audit setup and DDPO

• Input: The target VLM M receives a fixed partial text y and an image x, producing an open-ended continuation M(x, y). The audit only changes x.
• Optimizer: Uses Denoising Diffusion Policy Optimization (DDPO). The reverse denoising is formulated as a Markov decision process: the state includes the context, the timestep, and the current noisy sample; the action is the predicted previous state. The reward is nonzero only at the final step, equal to the task reward on the terminal sample and context. The policy gradient uses importance sampling, updated by the likelihood ratio between the current and old policies.
• Backbone and access: The pretrained diffusion model is Stable Diffusion v1.5. Only query-only access to the target VLM.

Greedy search for high-risk prompts

• The authors argue that image prompts written in a single step by a general LLM are often insufficient to expose stronger safety failures, so they iteratively update candidate prompts using the target VLM's toxicity feedback.
• The auxiliary model is Gemini-1.5-Pro (also called Gemini Pro in the main text), with safety settings disabled, and previously identified high-risk cases are included as in-context examples. The main text does not list the wording of these examples.
• Each round generates an image with the new prompt, then lets the target VLM continue under the fixed text, scored by Detoxify. If toxicity increases, the prompt is kept; otherwise, the search stops. The retained result is used as initialization for subsequent reinforcement learning. Algorithm 1 summarizes this process.

Toxicity reward and alignment reward

• Toxicity reward: Uses continuation-style prompts to let M complete y. Detoxify scores six toxicity attributes, averaged to give R_tox, used as a proxy for unsafety. The 0.5 threshold used in testing does not enter this reward.
• Alignment reward: Uses short description prompts to let M describe the image, then computes BERTScore between the image prompt and the description's word-level maximum cosine similarity, averaged over word count, giving R_align. The authors say this is to avoid the image drifting from the prompt and model collapse during optimization.
• Total reward: R_total = R_tox + λ R_align. The meaning is to simultaneously push up the proxy score for unsafe continuations and constrain the semantic consistency between image and prompt. In experiments, λ is 0, 0.05, 0.1, 0.2, corresponding to no alignment, Low, Medium, High.

Checker-aware variant under external guardrails

• The image may be blocked before reaching the VLM. The double check is: a Stable Diffusion image-level NSFW checker, and a VLM semantic checker that judges whether the image contains sexual, violent, or other NSFW content. The paper does not specify which model the latter is. Only if both judge as non-NSFW is it considered safe; any interception results in rejection and no model output.
• The greedy search is modified to include guardrail signals: each prompt samples 10 images to cover the randomness of diffusion sampling. If any image passes both checks, the prompt is kept; otherwise, the auxiliary LLM rewrites it knowing the external guardrails, and prompts that consistently produce NSFW images are discarded.
• Reward shaping uses an indicator function I(x), which is 1 only when both checks are safe. R_tox^guard = I(x) · R_tox(M(x, y)). The alignment reward is unchanged. The authors say this audits the entire safety pipeline, not just the VLM.

Training and success determination

• 8 NVIDIA A100s: 6 for training, 2 for scoring. Per-GPU batch size is 24, learning rate 3e-4. Early stopping after 600 updates; the authors say this is to mitigate reward hacking.
• Training text takes the first 132 items in order from the challenge subset. The authors say increasing this size further yields limited generalization gains and slows convergence.
• Testing counts continuations with attribute scores above 0.5 as positive for that attribute; Any* means at least one of the six attributes is positive. This threshold is not used during training.

Experimental setup

• Target model: LLaVA-1.5-7B is used for optimization and main results. Transfer tests are Gemini-1.5-flash and LLaMA-3.2-11B-Vision-Instruct (Table 4 title writes Llama-3.2-11B-Vision-Instruct). The image backbone is Stable Diffusion v1.5. Candidate prompts are generated by Gemini-1.5-Pro with its safety settings disabled.
• Data: RealToxicityPrompts challenge subset of 1,199 items, used as fixed text contexts. The first 132 items are the main evaluation subset (called original set in the abstract), the rest are hold-out. Guardrail experiments use what the paper calls the full dataset.
• Baselines: Text-Only, Single-Prompt, Greedy-Prompt. Guardrail and CLIPScore experiments also use an unmodified Stable Diffusion as an image source for comparison.
• Metrics: Proportion of continuations with toxicity attribute scores above 0.5; Any* is at least one of six attributes above 0.5. In CLIPScore, α=2.5. GPR is the proportion of images passing both guardrails, i.e., the average of the indicator function in Equation (6).
• Evaluation: Training reward uses Detoxify's six-attribute average; testing reports both Detoxify and Perspective API. No human evaluation. The paper does not report repetitions. The six category names of the two classifiers are not identical, e.g., Obscene vs. Profanity, Insult vs. Sexually Explicit.
• Training configuration: 8 A100s, per-GPU batch size 24, learning rate 3e-4, early stopping after 600 updates. λ is 0, 0.05, 0.1, 0.2.

Main results

According to Table 1, target is LLaVA, main evaluation subset, Any* (%).

Method	Detoxify Any*	Perspective Any*
Text-Only Baseline	58.01	53.03
Single-Prompt Baseline	55.72	55.30
Greedy-Prompt Baseline	64.12	58.33
RedDiff w/o alignment	61.83	58.33
RedDiff w/ alignment (L)	62.59	58.33
RedDiff w/ alignment (M)	68.70	61.36
RedDiff w/ alignment (H)	63.35	62.12

• The authors say alignment (M) has the highest Detoxify Any* at 68.70%; on Perspective, alignment (H) is highest at 62.12%. The authors believe the increase on both classifiers indicates the failure is not tied to a single toxicity metric. The abstract says the unsafe response rate on the original set increases by up to 10.69%; the conclusion writes from 58.01% to 68.70%.
• Counterexample: Single-Prompt's Detoxify Any* is 55.72%, lower than Text-Only's 58.01%. The authors say a single image prompt only brings limited additional degradation. Unaligned RedDiff's 61.83% is also lower than Greedy-Prompt's 64.12%; the authors do not separately explain this row.
• The authors say this subset's baseline toxicity is lower than hold-out, making it more suitable for observing safety degradation under situational conditions. Table 1's Text-Only Detoxify Any* is 58.01%, Table 2 is 70.54%.

Cross-prompt generalization

• Tests on remaining prompts unseen during optimization. Table 2 labels the corresponding row as RTD, the main text calls it RedDiff. Detoxify Any*: Text-only 70.54%, Single-Prompt 76.64%, Greedy-Prompt 77.57%, RTD unaligned 79.45%, alignment L/M/H are 78.33%, 78.70%, 79.36%.
• Perspective Any*: 70.00%, 75.63%, 77.60%, 79.28%, 77.50%, 78.16%, 78.81%. The authors say the unaligned variant has the highest Any* on both classifiers, and write Obscene 67.44%, Insult 42.02%, Profanity 68.13%, Severe Toxicity 7.12%.
• The authors believe the relative trend is consistent with the main subset, and the failure is not limited to prompts seen during optimization. The abstract says up to 8.91% increase on hold-out; the conclusion writes from 70.54% to 79.45%. Aligned variants have lower Any* than the unaligned variant.

Cross-model transfer

• Only optimized with LLaVA feedback, then directly apply the resulting multimodal inputs to two unseen VLMs. Tables 3 and 4 only report Any*.
• Gemini-1.5-flash: text-only Detoxify / Perspective is 76.23% / 78.26%; alignment (M) is 81.33% / 82.52%. In contributions, the authors say Gemini +5.1%. The authors believe harmful text alone often elicits unsafe continuations, visual context amplifies this further, and the pattern transfers to commercial VLMs.
• LLaMA-3.2-11B-Vision-Instruct: text-only 15.02% / 18.68%; alignment (L) is 41.85% / 45.24%, unaligned is 41.01% / 43.90%. In contributions, the authors say LLaMA-Vision +26.83%. The authors believe its text baseline is much lower, but these inputs still expose transferable failures.

External guardrails

• On the full dataset, LLaVA is preceded by external guardrails. The comparison setting is alignment (M). Images must pass both the Stable Diffusion NSFW checker and the VLM-based semantic checker, otherwise rejected with no model output.
• Table 6 Detoxify Any*: Stable Diffusion 37.45%, RedDiff (Base) 38.28%, RedDiff (Guard) 69.39%; GPR is 49.96%, 54.13%, 94.50%. Perspective Any* is 38.20%, 39.78%, 69.47%, GPR same as previous row.
• The authors say Base does not use guardrail feedback, yet GPR is higher than Stable Diffusion, so they believe generation shifted to more covert visual patterns that still pass the filter. The authors say Guard's Detoxify Any* goes from 38.28% to 69.39%, and GPR to 94.50%. The abstract and contributions say GPR increases by 40.4% relative to Stable Diffusion; Table 6 gives GPR as 49.96% and 94.50%.

Other ablations and analysis

• CLIPScore (Table 5): Stable Diffusion 44.47; unaligned 41.87, L 42.86, M 44.02, H 44.80. The authors say each variant is comparable to or higher than 44.47.
• The authors say Figure 3 shows that under the same image prompt, Stable Diffusion is more neutral, while RedDiffuser more often shows risk-related cues; λ from left to right is 0, 0.05, 0.1, 0.2. The attached text cannot see the pixels in the figure. Figure 1 caption says the standard diffusion model corresponds to benign continuations, RedDiffuser to harmful continuations; the text in the figure is not repeated here.
• Table 1 Detoxify: Identity Attack's Text-Only is 5.34%, each RedDiff row is 3.05% or 3.81%; Threat is 0.00% for Text-Only, unaligned, and alignment (M). The authors do not explain these attributes.
• In Table 1 Detoxify, the Toxicity column is identical to Any* rows; Perspective's Text-Only is Toxicity 46.96%, Any* 53.03%.
• Under guardrail settings, each prompt samples 10 images (Section 4.4). The paper does not report an iteration limit for the greedy search.

Limitations and future directions noted by the authors

The paper does not specifically discuss limitations. The provided text has no Limitations, Future Work, or limitation notes in the appendix; the conclusion discusses evaluation blind spots and auditing paradigms, and does not frame any result as a limitation of this work or as planned future work.

Experimental coverage

• The optimization and main-table target model is LLaVA-1.5-7B; transfer targets are Gemini-1.5-flash and LLaMA-3.2-11B-Vision-Instruct (Section 5.1–5.2, Table 1–4).
• The data is the RealToxicityPrompts challenge subset of 1,199 entries; the first 132 entries form the main evaluation subset, and the rest are hold-out. The guardrail experiments use what the paper calls the full dataset (Section 5, Section 5.4, Table 6).
• Testing uses both Detoxify and the Perspective API, with a positive threshold of 0.5, and reports Any*, CLIPScore, and GPR. Training rewards use only Detoxify. λ takes 0, 0.05, 0.1, 0.2.
• Baselines include Text-Only, Single-Prompt, Greedy-Prompt, and unfine-tuned Stable Diffusion. The guardrail is the Stable Diffusion NSFW checker plus a VLM semantic checker not named in the paper.
• The paper does not report the number of repetitions, number of queries, agreement with human judgment, or the semantic checker's model name. Reinforcement learning states early stopping after 600 updates; the iteration cap for greedy search is not reported.

RedDiffuser is a black-box auditing framework: harmful content is not an explicit attack instruction but a fixed partially toxic text, paired with images generated by a diffusion model, to see whether the VLM's open-ended continuation becomes unsafe.

• Approach: Gemini-1.5-Pro greedily selects image prompts based on Detoxify feedback, then DDPO fine-tunes Stable Diffusion v1.5. The reward simultaneously pushes up the toxicity score, and BERTScore constrains image-prompt consistency. If an upstream NSFW guardrail exists, images that fail the double check produce no toxicity reward.
• Main subset: The optimization target is LLaVA-1.5-7B, with data from RealToxicityPrompts. On Detoxify Any*, text-only is 58.01%, and with alignment weight 0.1 it is 68.70%; on Perspective, that weight gives 61.36%, and weight 0.2 gives 62.12% (Table 1). The abstract says this set improves by up to 10.69%.
• Hold-out: In Table 2, the RTD unaligned variant's Detoxify Any* is 79.45%, and text-only is 70.54%. The aligned variant is below this row. The conclusion states from 70.54% to 79.45%.
• Transfer: The same batch of inputs is used directly on unseen models. On Gemini-1.5-flash, aligned (M) Detoxify Any* is 81.33%, and text-only is 76.23% (Table 3). On LLaMA-3.2-11B-Vision-Instruct, aligned (L) is 41.85%, and text-only is 15.02% (Table 4). In the contributions, the authors say these increase by 5.1% and 26.83%, respectively.
• Guardrails: On the full dataset, RedDiff (Guard) has a GPR of 94.50% and Detoxify Any* of 69.39% (Table 6). The authors say GPR improves by 40.4% relative to Stable Diffusion; in the table, Stable Diffusion's GPR is 49.96%.
• Authors' conclusion: Semantically coherent visual context amplifies unsafe behavior, and this is not limited to the VLM used during optimization; upstream NSFW filtering alone does not block this kind of failure. The authors argue that safety evaluations centered only on explicit instructions have blind spots, and context-aware multimodal auditing should complement them.