Behind the mask: emergent security vulnerabilities in diffusion LLMs

1. 01What problem does this paper attempt to solve?

   The authors state that dLLMs' bidirectional modeling and parallel decoding render existing alignment ineffective, and present the first systematic jailbreak study targeting dLLMs.

   Core problem: the bidirectional context modeling and parallel decoding mechanisms of diffusion large language models (dLLMs) make existing safety alignment mechanisms unable to defend against context-aware masked-input attacks.

   • Problem setting: dLLMs support arbitrarily inserting mask tokens into instructions or generated content for editing, formatted generation and information extraction (Figure 1); the authors argue this capability also brings security risks.
   • Shortcomings of existing methods: the authors state that existing jailbreak attacks and defenses are almost all designed for autoregressive LLMs, and the jailbreak safety of dLLMs is largely unexplored; defenses designed for left-to-right models fail on dLLMs.
   • Core observation: bidirectional modeling forces the model to generate context-consistent content for masked spans (even if harmful), and parallel decoding prevents the model from dynamically filtering or rejection-sampling unsafe content during generation.
   • What the paper proposes: DIJA—which the authors describe as the first systematic study and jailbreak attack framework targeting dLLMs—automatically rewrites the original harmful prompt into an interleaved mask-text prompt; it also proposes a preliminary architecture-aware alignment strategy, Refusal-Aware Denoising Alignment (Appendix A).
   • Threat model (Appendix B.7):
     • Goal: induce harmful completions on an aligned dLLM.
     • Knowledge and capabilities: assumes the ability to remask or edit assistant-side responses; for open-source models, the original mask token can be injected directly.
     • Closed-source conditions: closed-source models require the provider to expose an editing or infilling API; otherwise, disabling arbitrary mask placement mitigates the attack.
     • Victim systems: publicly available dLLMs (the LLaDA, Dream, MMaDA and other families).

   Collapse this answer

2. 02What related research exists?

   Related work falls into three categories: diffusion language models, jailbreak attacks, and defenses; the authors state that dLLM jailbreak safety has not yet been systematically studied.

   Diffusion large language models

   • Masked Diffusion Models: Austin et al. (2021), Lou et al. (2023), Shi et al. (2024) and others generate text by iteratively predicting masked tokens, offering an alternative route to the autoregressive framework.
   • LLaDA (Nie et al., 2025b): an 8B-parameter bidirectional Transformer diffusion language model trained from scratch.
   • Dream (Ye et al., 2025): built on pretrained autoregressive weights; the paper states that both achieve performance comparable to autoregressive models such as LLaMA3 8B.

   Jailbreak attacks

   • Search/optimization-based: GCG (Zou et al., 2023), AutoDAN (Liu et al., 2023), PAIR (Chao et al., 2025) use optimization or genetic algorithms to generate adversarial prompts.
   • Side-channel and reasoning-exploitation-based: low-resource language attacks (Deng et al., 2023), scenario nesting (Ding et al., 2023), prompt decomposition (Li et al., 2024), ASCII obfuscation (Jiang et al., 2024).
   • Anthropomorphism-based: inducing harmful outputs through persuasion or cognitive overload (Li et al., 2023; Zeng et al., 2024; Xu et al., 2023).

   Jailbreak defenses

   • Four categories of defenses: filtering based on perplexity or external classifiers (Jain et al., 2023, etc.); input permutation or rewriting (Robey et al., 2023); prompt reminders (Xie et al., 2023); optimization-based robust prompts and safety alignment (Zhou et al., 2024; Xu et al., 2024, etc.).
   • Shortcoming noted by the paper: these methods were mainly developed for autoregressive LLMs, and the jailbreak safety of dLLMs is largely unexplored.

   Baselines and benchmarks

   • Attack baselines: Zeroshot, GCG, AIM, PAIR, ReNeLLM; defense baselines are Self-reminder and RPO.
   • Benchmarks: HarmBench, JailbreakBench, StrongREJECT.

   The authors position this paper as the first systematic study of dLLM safety issues, and state that DIJA is conceptually related to prefilling attacks on autoregressive models, but exploiting the non-autoregressive properties of dLLMs allows manipulation of arbitrary masked spans in the response (Appendix B.7).

   Collapse this answer

3. 03How does the paper address this problem?

   DIJA uses an auxiliary LLM to rewrite harmful prompts into interleaved mask-text prompts, forcing the dLLM to complete the harmful content at the masked positions.

   Overall approach: exploiting the two properties of dLLMs, bidirectional context modeling and parallel decoding, to automatically transform the original harmful prompt into a DIJA interleaved mask-text prompt—the harmful intent is retained in the unmasked fixed text, forcing the model to generate context-consistent harmful content at the masked positions.

   Problem formulation

   • Given a harmful prompt a and Q consecutive masks m, construct p_i = a ⊕ (m ⊗ w), where ⊕ is concatenation, ⊗ is interleaving, and w is harmless separator text; the construction process does not mask or delete the original harmful content.
   • The output distribution decomposes as P_ϕ(y|p_i) = ∏_{t ∈ M} P_ϕ(y_t|p_{i∖ t}) · ∏_{t ∉ M} δ(y_t = p_t): unmasked tokens are fixed, and masked tokens must be generated based on context, which the authors state makes it difficult for the model to refuse.

   Prompt construction (in-context learning)

   • Auxiliary model: Qwen2.5-7B-Instruct (denoted DIJA) or GPT-4o (denoted DIJA*) is used to automatically generate interleaved prompts through few-shot in-context learning; the template is in Appendix C.5.
   • Prompt diversification: manually curated examples covering various harmful forms (step-by-step guides, Q&A, lists, dialogue, etc.) and harmful categories, with style perturbations added.
   • Mask pattern selection: three strategies—Block-wise masking (masking whole spans), Fine-grained masking (selectively hiding key tokens), Progressive masking (incrementally increasing masking across multiple steps).
   • Harmless separator text insertion: obtained from a phrase pool or controlled generation, stylistically consistent, semantically neutral, no more than 10 words, used to maintain fluency and anchor the context.

   Attack procedure (Algorithm 1)

   1. Stage one: initialize the mask sequence, provide few-shot examples and the original harmful prompt to the auxiliary LLM, and generate and refine the interleaved prompt.
   2. Stage two: feed the refined prompt into the victim dLLM; masked positions are sampled and decoded in parallel, and unmasked positions are forced to keep their original tokens.

   Success criteria and evaluation methods

   • ASR-k: judged successful if the response contains none of the predefined refusal keywords (keyword list in Table 17).
   • ASR-e: HarmBench uses the fine-tuned HarmBench-Llama-2-13b-cls judge; JailbreakBench uses Meta-Llama-3-70B-Instruct per its guidelines.
   • HS: GPT-4o scores on a 1–5 scale, with HS=5 judged as a successful jailbreak; SRS: the StrongREJECT fine-tuned judge model scores, rescaled from [0,1] to [0,100].

   Preliminary defense (Appendix A)

   • Refusal-Aware Denoising Alignment: uses about 10,000 harmful instructions (5,000 from WildGuard + 5,000 from the Circuit Breaker Dataset) to generate adversarial prompts via DIJA, paired with standardized refusals, to fine-tune LLaDA-8B-Instruct to learn to output refusals to interleaved mask structures.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On 4 dLLMs and 3 benchmarks, DIJA's ASR-k is generally the highest, and the authors state it remains effective against both defenses.

   Experimental Setup

   • Victim models: LLaDA-8B-Instruct, LLaDA-1.5, Dream-v0-Instruct-7B, MMaDA-8B-MixCoT; Appendix B.1 adds three code dLLMs: DiffuCoder-7B-Instruct, DiffuCoder-7B-cpGRPO, Dream-Coder-v0-Instruct-7B.
   • Comparison autoregressive models: LLaMA-3.1-8B-Instruct, Qwen2.5-7B-Instruct (only for the defense comparison in Figures 3, 8, 9).
   • Benchmarks: HarmBench, JailbreakBench (containing 100 violating behaviors), StrongREJECT.
   • Baselines: Zeroshot, GCG, AIM, PAIR, ReNeLLM; defenses are Self-reminder, RPO.
   • Metrics and judges: ASR-k (keywords), ASR-e (HarmBench uses HarmBench-Llama-2-13b-cls, JailbreakBench uses Meta-Llama-3-70B-Instruct), HS (GPT-4o scores 1–5, HS=5 is success), SRS (StrongREJECT fine-tuned judge).
   • Hyperparameters: gen_length=512, block_length=32, steps=32, temperature=0.2; stability experiments run 3 independent times (Appendix B.4).

   Main Results

   According to Table 1 (HarmBench), Table 2 (JailbreakBench), Table 3 (StrongREJECT), the ASR-k / ASR-e / HS or SRS of each model under DIJA* (constructed by GPT-4o):

   Swipe horizontally to read the table

   Victim model	HarmBench ASR-k	HarmBench ASR-e	JBB ASR-k	JBB ASR-e	SR ASR-k	SR SRS
   LLaDA-Instruct	98.0	60.0	99.0	81.0	99.7	62.4
   LLaDA-1.5	99.3	58.8	100.0	82.0	99.4	63.3
   Dream-Instruct	99.0	60.5	100.0	88.0	99.7	52.2
   MMaDA-MixCoT	99.0	47.3	100.0	81.0	99.0	47.6

   • The authors state that DIJA achieves the highest ASR-k on all tested dLLMs across the three benchmarks, with some models reaching 100%; on Dream-Instruct, ASR-e is 54% higher than the second-place ReNeLLM (HarmBench) and 78.5% higher (JailbreakBench), and SRS is 37.7 higher (StrongREJECT).
   • The authors state that DIJA* is slightly better than DIJA, attributed to GPT-4o's stronger few-shot and instruction-following capabilities.
   • Exceptions: on HarmBench, ReNeLLM's ASR-k against LLaDA-Instruct is 98.0, on par with DIJA*; on StrongREJECT, DIJA (Qwen version)'s ASR-k against LLaDA-Instruct is 92.7, lower than ReNeLLM's 93.3.

   Robustness to Defenses

   • What was tested: DIJA was evaluated under two defenses, Self-reminder and RPO (Figure 5, three victim dLLMs × three benchmarks).
   • Results: The bar heights in the figure show that ASR-e/SRS with defenses are close to those without defenses; the authors state that attack effectiveness does not drop significantly (specific values in the figure are not labeled).
   • Authors' interpretation: DIJA is robust to these two commonly used defenses.

   Preliminary Alignment Defense (Appendix A)

   • What was tested: LLaDA-8B-Instruct was fine-tuned with about 10,000 alignment data to obtain LLaDA-Instruct-Aligned, then DIJA was run.
   • Results: On JailbreakBench, ASR-k drops from 95.0 to 19.0 and ASR-e from 81.0 to 25.0; HarmBench ASR-e drops from 55.5 to 32.5; StrongREJECT SRS drops from 60.8 to 29.4 (Table 4).
   • Utility cost: GSM8K goes from 78.5 to 78.9, MMLU from 65.7 to 64.2, GPQA from 32.4 to 27.5 (Table 5); the authors state that the utility cost of the safety improvement is modest.

   Analysis of Mask Count and Generation Length

   • Generation length: Figure 6 shows that LLaDA-1.5 refuses at generation length 256 and produces harmful content at 512; the authors hypothesize that bidirectional attention makes responses affected by the allowed generation length.
   • Number of mask tokens: Figure 7 shows that with 10 masks the attack effectiveness on the four dLLMs is limited, and with 50 masks ASR, SRS, HS often decrease; the authors state that overly long generation introduces irrelevant content.

   Other Ablations and Analyses

   • w/o Refinement LLM (Table 11, LLaDA-8B-Instruct, HarmBench): ASR-e 54.8%, close to the full DIJA's 55.5%.
   • w/o Masking: ASR-e 14.0%; w/o Separators: ASR-e 16.8% (Table 11).
   • Number of mask segments (Table 12, total budget 50 tokens): ASR-e is 31.5% with 1 segment, 55.5% with 3 segments, and drops to 43.5% with 10 segments.
   • Code dLLMs (Tables 6–8): DIJA raises ASR-k on all three code models to 94.3%–98.8%.
   • Stability (Table 10): ASR standard deviation across 3 runs < 0.7%.
   • Human verification (Table 18, n=100): agreement between GPT-4o and human binary classification is 93.0%–94.0%, Cohen's κ 0.86–0.88.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list directions such as multimodal dLLMs, defense generalization and utility cost; the experiments cover only open-source dLLMs and two defenses.

   Limitations and Future Directions Noted by the Authors

   • Multimodal and unified dLLMs: Explore the safety of multimodal and unified dLLMs in image and video generation and multimodal understanding tasks (Appendix E).
   • Defense generalization: Evaluate the generalization and robustness of refusal-aware alignment to unseen or modified jailbreak patterns (Appendix E).
   • Quantifying utility cost: Quantify the utility degradation of alignment on more benign tasks, especially tasks involving interleaved mask-text prompts (Appendix E).
   • Privacy and watermarking: Privacy risks and watermarking defenses in dLLMs are worth exploring (Appendix E).
   • Closed-source interface prerequisite: The authors note that for closed-source models, the attack requires the provider to expose an editing or infilling API; otherwise, disabling arbitrary mask placement would mitigate it (Appendix B.7).

   Scope of Experimental Coverage

   • The tested victim models are 7 open-source dLLMs (4 general-purpose, 3 code models), plus 2 autoregressive models used only for the defense comparison (Figures 3, 8, 9).
   • Defense evaluation includes only two defenses, Self-reminder and RPO; the paper states that perplexity-based defenses were not included because dLLM inputs require concatenating mask tokens, which would artificially inflate perplexity (Appendix C.4).
   • The benchmarks are HarmBench, JailbreakBench, and StrongREJECT; the stability analysis runs 3 independent times, and the paper does not report the number of repetitions for the other experiments.
   • The human verification sample size is 100 responses (25 per victim dLLM), scored by 2 evaluators (Appendix D.4).
   • The paper does not report the number of queries or the iteration budget of the attack; the alignment experiment is conducted on only one model, LLaDA-8B-Instruct (Appendix A).

   Collapse this answer

6. 06Summarize the main content of the paper

   DIJA exploits dLLMs' bidirectional modeling and parallel decoding to carry out jailbreaks; the authors state it achieves the highest ASR-k across multiple benchmarks and present a preliminary defense.

   A jailbreak attack paper targeting diffusion large language models (dLLMs), which the authors describe as the first systematic study of dLLM safety issues.

   • Problem: the bidirectional context modeling of dLLMs forces the model to generate context-consistent content for masked spans, and parallel decoding prevents it from dynamically refusing during generation; the authors argue that existing alignment mechanisms designed for autoregressive models therefore fail.
   • Method: DIJA uses an auxiliary LLM (Qwen2.5-7B-Instruct or GPT-4o) with in-context learning to rewrite the original harmful prompt into an interleaved mask-text prompt, without rewriting or hiding the harmful content, forcing the model to complete the harmful instruction at the masked positions.
   • Main results: on LLaDA-8B-Instruct, LLaDA-1.5, Dream-Instruct, MMaDA-MixCoT and HarmBench, JailbreakBench, StrongREJECT, the authors state that DIJA's ASR-k is the highest on all tested dLLMs, with multiple DIJA* settings reaching 99%–100% (Tables 1–3); on Dream-Instruct, ASR-e is 78.5% higher than ReNeLLM (JailbreakBench), and SRS is 37.7 higher (StrongREJECT).
   • Robustness and ablation: remains effective against the two defenses Self-reminder and RPO (Figure 5); after removing the auxiliary LLM, ASR-e is still 54.8%, on which basis the authors argue the vulnerability stems from the interleaved mask structure itself (Table 11).
   • Preliminary defense: Refusal-Aware Denoising Alignment fine-tunes LLaDA-8B-Instruct with about 10,000 data items, reducing ASR-e on JailbreakBench from 81.0 to 25.0 (Table 4), with little change in general capabilities (Table 5).
   • Authors' conclusion: dLLMs are not unalignable, but require architecture-aware, customized alignment strategies; the authors call on the community to pay attention to this overlooked attack surface.

   Collapse this answer

Related papers