LLMs are vulnerable to malicious requests whose intent is hard to distinguish
LLMs are Vulnerable to Malicious Requests with Indistinguishable Intent
The authors propose the DGA jailbreak framework, which uses an auxiliary LLM to expand malicious intent into detailed requests, achieving an average ASR of up to 97.22% across 11 models and 3 datasets (AdvBench, DGA-OS).
black-box single-turn jailbreak: the attacker can only query the target LLM through inputs and outputs, cannot access model internals, training data, or system prompts, and cannot modify weights, filters, or the deployment pipeline; each constructed request is submitted only once, and the goal is to induce the model to generate harmful or policy-violating replies.
- Jailbreak attacks can bypass LLM safety mechanisms. The authors first evaluate eight jailbreak methods with a filter based on a defense prompt, and find that jailbreaks that resemble normal requests (such as PTA and SS) have a pass rate far higher than jailbreaks with obvious anomalous patterns, from which they propose the hypothesis: similarity to normal requests may contribute to jailbreak success.
- Proposes Detail Guidance Attack (DGA): an auxiliary LLM rewrites a malicious request into a noun phrase, then generates details around it—observable signs (DGA-OS) or a corresponding story (DGA-CS)—concatenates an induced prompt, and submits it to the victim model in a single turn; it also evaluates a filter containing knowledge of DGA patterns and conducts a 95-person user study.
- Across 11 models and 3 datasets, DGA-OS has an average ASR of 95.27% (MI), 93.82% (JBB), and 97.22% (AdvBench), higher than the eight baselines; Self-Reminder and Llama-Guard-3-8B have almost no defensive effect; the DGA-aware filter can reduce the PR of DGA-CS on MI to 8% (o4-mini), but its false positive rate was not measured.
- The authors state: the experiments did not separate the respective contributions of details, scenario framing, and response constraints, and did not verify that similarity is a causal mechanism; the DGA-aware filter's false positive rate on benign requests was not measured; IS is a model score and was not validated against human judgment; the user study only records preferences and does not evaluate content safety.
- Evaluated models
- GPT-4oGPT-5o3o4-miniGemini-2.5-flashQwen-3-235B-A22BDeepSeek-v3GLM-4.5Doubao-Seed-1.6Kimi-K2-0711-previewStep-2-16K
- Benchmarks
- MaliciousInstructJBB-BehaviorsAdvBenchXSTest
- Metrics
- ASRPRASRfharmfulness score (1–5)ISMLS
Wenzhuo Xu and other researchers propose the Detail Guidance Attack (DGA) jailbreak framework, which uses an auxiliary LLM to construct highly detailed requests and conducts single-turn interactions with the victim model. The authors first test eight jailbreak methods on an LLM-based filter and observe that requests similar to normal user queries are more likely to pass filtering, based on which they hypothesize that similarity to normal requests may contribute to jailbreak success. Across 11 victim models and three datasets, the full DGA pipeline achieves high attack success rates under their automated judging criteria, exceeding 95% in multiple evaluation settings. The authors also evaluate filters capable of identifying DGA, and conduct a user survey on everyday requests with ambiguous intent or dual use; the survey shows that people hold differing expectations about whether models should respond, and the usability cost of filters remains to be measured.
Paper analysis
What problem does this paper attempt to solve?
The authors hypothesize that "resembling a normal request" facilitates jailbreak success and propose the DGA framework for single-turn black-box jailbreaking using detailed requests.
Core question: can malicious requests that are hard to distinguish from normal user requests bypass LLM safety mechanisms, and why current filters fail to block such requests.
- Problem setting: jailbreak attacks can bypass LLMs' built-in safety mechanisms and induce them to generate harmful content. The authors first build an LLM filter with a defense prompt and evaluate it with eight jailbreak methods, finding that the filter pass rates of different methods vary greatly.
- Shortcomings of existing methods (as stated in the paper): the authors observe that jailbreaks containing obvious anomalous patterns (such as DMA, CA) have pass rates close to 0%, whereas jailbreaks that resemble normal requests (PTA, SS) have average pass rates above 50%; on this basis the authors divide jailbreaks into Type I (containing obvious attack patterns) and Type II (appearing like ordinary queries) (Section 4.3).
- Core hypothesis: similarity to normal requests may help bypass filters and achieve jailbreak success; detailed narration is one candidate way of producing this similarity, motivated by research showing that users provide contextual information when seeking help (Section 5).
- What the paper proposes: the Detail Guidance Attack (DGA) framework, which uses an auxiliary LLM to expand a concise malicious intent into a detailed request and interacts with the victim model in a single turn; it also evaluates a DGA-aware filter and conducts a user study on everyday requests with ambiguous intent.
- Threat model (Section 3.1, terms as given):
- Attack goal: construct requests that induce the LLM to generate harmful or policy-violating responses.
- Knowledge: black-box access, able only to query the target LLM and observe responses, with no access to model internals, training data, or system prompts.
- Capability: single-turn interaction, each request submitted only once, unable to modify model weights, safety filters, or the deployment pipeline; request construction may call auxiliary models.
- Victim system: 11 modern LLMs, with the filter and the victim model being two independent copies of the same LLM.
What related research exists?
The paper divides related work into two groups, jailbreak attacks and jailbreak defenses, and positions itself as a scenario-based attack that dynamically generates details.
Jailbreak attacks
- Gradient and mutation categories: gradient-based jailbreaks [37,16], mutation-based jailbreaks [9,10]; the paper mentions these only categorically.
- Prompt categories (focus of this paper): CipherChat [35] encodes harmful requests into ciphers to bypass safety constraints; DeepInception [15] creates virtual nested scenarios to cross moral boundaries; Past Tense Attack [2] uses past-tense rewriting for jailbreaking. The paper states that such methods require no access to model internals and are easy to execute.
- Scenario and role-play categories: persona-modulation [28] and Persuasive Adversarial Prompts (PAP) [36]. The paper states that its method is similar to scenario simulation or role-playing, but differs in that details are dynamically generated rather than using fixed templates, and the details target the original task and incorporate response constraints (Section 2.1).
Jailbreak defenses
- Detection and filtering categories: PPL [1] blocks by perplexity threshold; the Llama-Guard and GA Guard series [19,7] are specially fine-tuned harmful-content detection models; GradSafe [31] identifies harmful requests based on anomalous gradient patterns.
- Prompt defense categories: Self-Reminder [32] uses safety prompts to remind the LLM to generate harmless replies. The paper states that Self-Reminder, Llama-Guard, and GA Guard were chosen to evaluate its method because they are suitable for black-box defense (Section 2.2).
Baselines and benchmarks
- Baseline attacks: DeepInception (DI), Developer Mode Attack (DMA), CodeAttack (CA), CipherChat (CC), FlipAttack (FA), Past Tense Attack (PTA), and two Scenario Simulation scenarios (Safety Training, Novel Writing), eight in total (Section 4.2).
- Baseline defenses: Self-Reminder, Llama-Guard-3-8B, GA Guard Lite (Section 6.2).
- Benchmarks/datasets: MaliciousInstruct (100 harmful requests), JBB-Behaviors (100), AdvBench (520); XSTest was used for threshold selection (Appendix D).
Positioning: The authors state that this paper is close to scenario simulation/role-play attacks, with the difference that details are dynamically generated by an auxiliary LLM according to the original task and combined with response constraints, rather than applying fixed templates.
How does the paper address this problem?
First build an MLS filter to observe jailbreak patterns, then use an auxiliary LLM to generate detailed requests for single-turn jailbreaking, and provide a DGA-aware defense.
The overall approach has three steps: first, use an LLM filter to evaluate eight jailbreak methods and observe which types of requests pass easily; then, based on this, propose Detail Guidance Attack (DGA), which uses an auxiliary LLM to expand malicious intent into detailed requests; finally, construct a filter containing knowledge of DGA patterns as a possible defense.
Filter and MLS (Section 4.1)
- Defense prompt: Contains only a task description ("You are a motivation classifier...") and scoring criteria, reminding the user that requests may solicit harmful output, but does not include specific jailbreak patterns; the full prompt is in Appendix B.
- MLS definition: Motivation Level Score is a 5-level score from 0–4, where 0 means definitely harmful and 4 means definitely harmless; the filter also outputs the reason for the score.
- Blocking rule: Requests are blocked when MLS is below the threshold; the threshold is 2, based on a trade-off between PR 93.05% on XSTest (benign) and PR 1.54% on MaliciousInstruct (malicious) in Appendix D.
Observations and classification (Section 4.3)
- Two types of jailbreaks: Type I contains obvious attack-related patterns (e.g., explicitly overriding previous instructions); Type II looks like an ordinary query (e.g., historical questions, scenario-based requests). The authors state that this classification concerns only request appearance and does not verify indistinguishability to human observers.
- IS score: An LLM scores the similarity between jailbreak requests and normal requests from 0–4 (indistinguishability score; prompt in Appendix E); Figure 1 gives the average scores of the eight baselines (e.g., PTA 2.48, CA 0.33). The authors state that methods with high IS generally also have high PR, but note that this is only a cross-method correlation, not a causal test.
DGA pipeline (Section 5)
- Rewriting: The original prompt Po is transformed by a rewriting function R() into a noun phrase Pk (e.g., "How can someone manufacture illegal drugs?" → "manufacturing illegal drugs"); the rewriting prompt is in Appendix F.
- Detail generation: The auxiliary LLM acts as a detail generator G(), generating a description Pd = G(Pk; t) according to detail type t, i.e., generating detail text around the key phrase.
- Two variants: DGA-OS generates observable signs related to Pk (Observable Signs); DGA-CS generates a corresponding story around Pk (Corresponding Story).
- Concatenation and submission: The final request is Pd ⊕ Pi, i.e., the details concatenated with the induced prompt and submitted to the victim LLM. For DGA-OS, Pi asks the victim model to infer possible steps from the signs; for DGA-CS, the clarification request and response constraints are embedded in the story template, and Pi is an empty string. Both variants contain response instructions; prompts are in Appendix G.
- Authors' note: The experiments evaluate the complete pipeline and do not separate the contribution of details from the scenario framing and response constraints.
Success judgment and evaluation (Sections 3.2, 6.1)
- Judgment criterion C(·): GPT-4 classifies replies as safe/unsafe using the response-only judgment prompt in Appendix A, with unsafe recorded as a successful attack; even if a reply includes a disclaimer or is presented in hypothetical or fictional form, it is judged unsafe as long as the information could be misused.
- Metrics: ASR (success rate without defense), PR (filter pass rate), ASRf (success rate with the filter); the 1–5 harmfulness score from [30] is also used.
- Implementation details: Gemini-2.5-flash serves as the detail generator; Gemini-2.5-flash is used for rewriting, falling back to GPT-4o on refusal (fewer than ten times in total across the three datasets) (Appendix F).
DGA-aware defense (Section 6.3)
- Construction: Collect several DGA-OS/CS jailbreak requests, use GPT-4o to analyze their features and generate a defense prompt containing descriptions of these two attack patterns (Appendix I).
- Filter: o4-mini and Qwen-3 serve as this jailbreak-strategy-aware filter, and PR is measured on the three datasets.
What experiments did the paper conduct?
Across 11 models and 3 datasets, DGA-OS achieves an average ASR of up to 97.22%; common defenses are largely ineffective, and the DGA-aware filter is effective but its false positive rate was not measured.
Experimental setup
- Models tested (11): GPT-4o, GPT-5, o3, o4-mini, Gemini-2.5-flash, Qwen-3-235B-A22B, DeepSeek-v3, GLM-4.5, Doubao-Seed-1.6, Kimi-K2-0711-preview, Step-2-16K; the filter and the victim model are two independent copies of the same LLM (Section 4.2).
- Datasets: MaliciousInstruct (100 harmful requests), JBB-Behaviors (100), AdvBench (520); XSTest is used for threshold selection (Appendix D).
- Baseline attacks: CA, CC, DI, DMA, PTA, SS-ST, SS-NW, FA, eight in total; baseline defenses are Self-Reminder, Llama-Guard-3-8B, and GA Guard Lite.
- Metrics and judging: ASR uses GPT-4 with the Appendix A prompt for response-only binary classification; PR is the filter pass rate (threshold MLS≥2); harmfulness uses the 1–5 scale of [30].
- Auxiliary models: Gemini-2.5-flash generates details and rewrites, falling back to GPT-4o when rewriting refusals (Appendix F).
- Sample size and repetitions: each method × model combination is run once on the full corresponding dataset; the paper does not report the number of repetitions.
Main results
Average ASR of DGA and the eight baselines across 11 models (%, from Table 2, Table 6, Table 7):
Swipe horizontally to read the table
Method MI (ASR) MI (PR) JBB (ASR) Adv (ASR) DGA-OS 95.27 95.55 93.82 97.22 DGA-CS 91.34 87.45 82.27 86.85 PTA (one of the highest baselines) 56.55 75.09 58.82 67.81 CA 61.27 2.09 43.64 38.76 CC 49.36 15.55 50.18 49.06 DI 52.36 3.45 39.91 43.67 SS-NW 54.11 49.64 46.18 40.37 DMA 10.00 0.18 18.82 10.89 - The authors state that the jailbreak performance of DGA-OS/CS exceeds the compared attacks by a large margin: the highest baseline average ASR is 61.27% (CA, MI), and the highest average PR is 75.09% (PTA, MI).
- The authors also state that these results characterize the performance of the complete DGA pipeline under the judgment criteria used, do not separate the role of details, and do not establish an improvement in indistinguishability.
- By model (Table 6), DGA-OS on MI has the lowest ASR of 88% on Qwen3 and the highest of 100% on Seed-1.6 and Kimi-k2; DGA-CS on GPT-5 has ASR 76% and PR 42%, the relatively weaker row for this variant.
Evaluation against defenses (Table 9)
- Self-Reminder (on Gemini-2.5-flash): DGA-OS ASR is 97 (MI), 89 (JBB), 98.37 (Adv.), decreases of 1, 0, and 1.25 percentage points relative to no defense; DGA-CS decreases by 1, 2, and 2.69 percentage points. The authors state that Self-Reminder has almost no defensive effect against DGA-OS/CS.
- Llama-Guard-3-8B: PR for DGA-OS is 100 (MI), 93 (JBB), 92.88 (Adv.); the authors state its defensive effect is very weak.
- GA Guard Lite: PR for DGA-OS is 100 (MI), 100 (JBB), 99.81 (Adv.), i.e., it essentially does not intercept; PR for DGA-CS is 69 (MI), 58 (JBB), 63.84 (Adv.); the authors state it can intercept a moderate proportion of DGA-CS requests but has no defense against DGA-OS.
DGA-aware filter (Table 10)
- What was tested: GPT-4o is used to analyze DGA request features and generate a dedicated defense prompt, with o4-mini and Qwen-3 serving as filters, measuring PR on DGA requests.
- Results: o4-mini PR for DGA-OS is 15 (MI), 8 (JBB), 3.85 (Adv.), and for DGA-CS is 8 (MI), 10 (JBB), 6.73 (Adv.); Qwen-3 for DGA-OS is 25/28/19.23, and for DGA-CS is 23/7/7.3.
- Authors' interpretation: the filter can effectively intercept DGA-OS/CS requests; however, the authors explicitly state that its false positive rate on benign requests was not measured, and the current results cannot establish its usability cost (Section 6.4).
Harmfulness scores and user study
- Harmfulness (Table 8): DGA-OS averages 4.62/3.92/4.47 across 11 models on MI/JBB/Adv, and DGA-CS 4.68/3.98/4.30 (1–5 scale). The authors note that the table has no baseline scores, so no conclusion can be drawn that it is more harmful than the baselines.
- User study (Table 11, 95 valid questionnaires): 7 everyday requests with ambiguous intent, each with three preferences to choose from; the highest proportion favoring a response is 56.9% for Identity Theft, the lowest is 9.5% for Pornography; the highest proportion favoring refusal is 72.6% for Pornography. In addition, 66.3% of respondents reported having encountered a request they considered reasonable being refused. The authors state that these preferences concern only whether the model should respond, and that self-reported experiences cannot estimate the filter's false positive rate.
Other ablations and analyses
- IS scores (Figure 1): average IS of the eight baselines is PTA 2.48, NW 1.51, ST 1.36, DI 1.25, CC 0.41, CA 0.33, FA 0.19, DMA 0.10; the authors state that methods with high IS generally also have high PR, but this is only a correlation.
- Threshold selection (Appendix D, Table 23): at threshold 2, XSTest PR is 93.05% and MI PR is 1.54%, so threshold = 2 is used throughout.
- Qualitative examples (Figure 3): screenshots of responses elicited by DGA-OS on GPT-4o, GPT-5, Qwen-3, and Gemini-2.5-flash.
- Exceptions: DGA-CS is relatively weak on GPT-5 and o3 (ASR on JBB is 66 and 45, respectively, Table 30); the paper gives no explanation.
What are some points that can be further explored?
The authors explicitly state that they did not separate the contributions of components, did not verify the causal mechanism, and did not measure the filter's false positive rate; the experiments cover 11 models and 3 datasets.
Limitations and future directions noted by the authors
- Component contributions not separated (Section 5, 6.2, Conclusion): the experiments evaluate the complete DGA pipeline and do not separate the independent contribution of rich details from scenario framing and response constraints; a controlled evaluation is needed.
- Causal mechanism not verified (Section 4.3, Conclusion): the relationship between IS and PR is only a cross-method correlation, not a causal test; the IS results do not include DGA and were not validated against human similarity judgments.
- Filter false positive rate not measured (Section 6.4, Conclusion): the DGA-aware filter's false positive rate on benign requests was not measured, and neither its usability cost nor the "inevitable trade-off" is established.
- Limitations of ASR (Section 3.2): the judgment criteria include information that "could potentially be misused"; ASR itself cannot establish the actual operability or correctness of the responses.
- Boundaries of the user study (Section 7): preferences concern only whether the model should respond, and do not determine which content is safe or useful; self-reported refusal experiences cannot estimate the filter's false positive rate.
- Future directions (Section 6.4): attack interception and benign-request usability need to be evaluated together; whether additional signals such as historical context are beneficial remains to be evaluated.
Scope of the experiments
- The models tested are GPT-4o, GPT-5, o3, o4-mini, Gemini-2.5-flash, Qwen-3-235B-A22B, DeepSeek-v3, GLM-4.5, Doubao-Seed-1.6, Kimi-K2-0711-preview, and Step-2-16K, 11 in total (Section 4.2).
- The datasets are MaliciousInstruct (100 items), JBB-Behaviors (100 items), and AdvBench (520 items); XSTest is used for threshold selection (Section 6.1, Appendix D).
- The defense evaluation includes Self-Reminder, Llama-Guard-3-8B, GA Guard Lite, and the DGA-aware filter; Self-Reminder is tested only on Gemini-2.5-flash (Table 9 note).
- The auxiliary model for attack construction is Gemini-2.5-flash (detail generation and rewriting), falling back to GPT-4o when rewriting refusals; the ASR judge model is GPT-4 (Section 3.2, 6.1).
- The paper does not report the number of repetitions per condition, nor the agreement between judge results and human judgment.
Summarize the main content of the paper
DGA uses detailed requests for single-turn black-box jailbreaking, with an average ASR of up to 97.22%; the authors state that the similarity hypothesis still awaits controlled verification.
A jailbreak attack paper: draws observations from filter evaluation, proposes a detailed-request attack, and includes defense attempts and a user study.
- Problem addressed: The authors first evaluate eight jailbreak methods with an LLM filter that has a defense prompt (MLS score, threshold 2), finding that methods containing obvious anomalous patterns (CA, DMA) have an average PR close to 0%, whereas methods resembling normal requests (PTA average PR 75.09%, SS-ST 52.55%) have much higher pass rates (Table 2); they therefore hypothesize that similarity to normal requests may facilitate jailbreaking.
- Method highlights: DGA uses an auxiliary LLM (Gemini-2.5-flash) to rewrite malicious intent into a noun phrase, then generates observable signs (DGA-OS) or a corresponding story (DGA-CS), concatenates induced content containing response constraints, and submits it to the victim model in a single turn; the threat model is black-box, single-turn, with no modification capability.
- Main results: Across 11 models and 3 datasets, DGA-OS achieves average ASR of 95.27% (MI), 93.82% (JBB), and 97.22% (AdvBench), higher than the eight baselines (baseline maximum 61.27%, CA on MI); Self-Reminder reduces ASR by at most 2.69 percentage points, Llama-Guard-3-8B achieves a PR of 100% on DGA-OS (MI), and GA Guard Lite also largely fails to block DGA-OS (Table 9).
- Defenses and study: A filter containing knowledge of DGA patterns (o4-mini) reduces the PR of DGA-CS on MI to 8%, but the authors explicitly did not measure the false positive rate on benign requests; a 95-person user study shows that for everyday requests with ambiguous intent, respondents' preferences are dispersed (the proportion leaning toward a reply ranges from 9.5% to 56.9%).
- Authors' conclusions and implications: The authors state that the results characterize only the performance of the complete pipeline under automated judgment criteria, that they do not isolate the contribution of details, and that they do not establish similarity as a causal mechanism; they also argue that safety and usability for normal users need to be evaluated together.
Related papers
- Safety Alignment Should Be Made More Than Just a Few Tokens DeepAnalysis / interpretabilityICLR 2025
- The Struggle Between Continuation and Refusal: A Mechanistic Analysis of the Continuation-Triggered Jailbreak in LLMsAnalysis / interpretabilityarXiv