Paper proposes adaptive attack framework that breaks through 12 LLM jailbreak and prompt injection defenses

Current evaluation methods for safety defenses in large language models have flaws; reliance on static test sets or weak attacks not adapted to the defenses leads to systematic overestimation of defense robustness.

• Scenario and importance of the problem: In defenses against jailbreaks (where users induce the model to output harmful content in violation of safety alignment) and prompt injection (where third-party input induces the model to perform unauthorized actions against user intent), accurately assessing defense strength is crucial for measuring the safety of large model systems.
• Shortcomings of existing evaluation methods: The authors point out that current defense evaluations typically only reuse fixed datasets of known attack prompts, or directly apply general optimization algorithms without targeted adaptation (such as unadjusted GCG), and allocate low computational budgets.
• Core observation and hypothesis: Drawing on experience from adversarial machine learning and traditional security fields, the authors believe that only an adaptive attacker that explicitly adjusts attack strategies against the defense mechanism and is not artificially limited in compute can measure the intrinsic strength of a defense.
• Proposed approach in the paper: The paper proposes a unified adaptive attack framework covering four types of attacks—gradient-based, reinforcement learning, evolutionary search, and human red teaming—to systematically evaluate and test 12 recent representative defense mechanisms.
• Threat model:
  • Attack objective: For jailbreak attacks, induce the model to output publicly harmful content; for prompt injection attacks, achieve exfiltration of private data or compromise system integrity through tool calls.
  • Attacker knowledge: Covers three settings: Whitebox with full knowledge of architecture and parameters, Blackbox (with logits) with access to output probabilities of each token, and Blackbox (generation only) with access only to discrete generated text.
  • Attacker capability: The attacker knows the design details of the defense mechanism, can input arbitrary prompts to the target system, or can inject untrusted data into the results returned by tool calls.
  • Computational resources: Assumes the attacker has ample computational resources (e.g., running a GPU cluster for 24 hours) and can perform large-scale iterative optimization.

Related work mainly revolves around historical experience in adversarial machine learning, automated attacks on large models, and human red teaming.

• Experience with adaptive evaluation in adversarial machine learning:

  • Athalye et al. (2018) and Carlini & Wagner (2017a) — in the image adversarial example domain, pointed out that obfuscated gradients create a false sense of robustness, and adaptive attacks tailored to specific defenses broke many defenses.
  • Croce & Hein (2020) and Tramèr et al. (2020) — proposed parameter-free attack ensembles and adaptive norms for defense evaluation, noting that fixed test sets are prone to overfitting.
  • Privacy and backdoor defense evaluations (Aerni et al., 2024; Qi et al., 2023) — noted that in membership inference and backdoor defenses, the lack of adaptive attacks similarly leads to misjudgments of defense effectiveness.

• Automated jailbreak and prompt injection attacks on large models:

  • Gradient-based discrete optimization (Zou et al., 2023; Guo et al., 2024) — proposed methods like GCG and COLD, mapping continuous gradients to discrete token space to find adversarial suffixes, but with high computational cost and sensitivity to hyperparameters.
  • LLM-assisted and search-based attacks (Chao et al., 2023; Mehrotra et al., 2023; Andriushchenko et al., 2024) — use LLMs to generate or rewrite candidate prompts, performing tree search or iterative optimization in black-box settings.
  • Prompt injection attack research (Greshake et al., 2023; Perez & Ribeiro, 2022; Zhan et al., 2025) — formalized the indirect prompt injection threat and proposed adaptive bypass methods.

• Human red teaming and crowdsourced security challenges:

  • Crowdsourced attack-defense competitions (Schulhoff et al., 2023; Toyer et al., 2023) — collected human red team attacks through platforms like HackAPrompt and Tensor Trust, showing human flexibility in discovering complex jailbreaks.
  • Frontier model red team evaluations (Ahmad et al., 2025; Anthropic, 2025) — OpenAI and Anthropic organized human red team evaluations before model deployment to identify frontier risks.

• Baseline methods and benchmark datasets:

  • Evaluation benchmarks include HarmBench for testing jailbreaks, and AgentDojo, OpenPromptInject, and Alpaca (including the davinci_003 task) for testing prompt injection.
  • Comparison baselines include static attack prompts originally used by 12 defenses (e.g., AgentDojo's static important instruction templates) and weak non-adapted optimization attacks.

• Positioning of this paper: The authors position this work as systematically transferring the core principles of adaptive evaluation from adversarial machine learning to the security of large models, proposing a unified PSSU attack framework and combining large-scale compute with human red teaming to re-examine existing defenses.

The authors propose a unified general adaptive attack framework that formalizes the attack process as a multi-round iterative optimization loop and constructs four specific attack methods.

Formalization of the general adaptive attack loop

The attack process is organized as an iterative loop consisting of four steps: Propose, Score, Select, and Update (Figure 2).

1. Loop mechanism: In each optimization round, the attacker generates candidate inputs through the proposal module, then uses a scoring function to evaluate the target system's feedback, selects high-potential candidates, and uses them to update the attack generation strategy.
2. Core definition: The scoring step quantifies attack performance through a target oracle and a judgment function, and its optimization objective can be expressed as:

max_{x ∈ 𝒳} J(x, 𝒪(x)) where x is the input adversarial prompt or injected content, 𝒪(x) is the target system output, and the function J is used to evaluate attack success rate, detectability, and cost.

Reinforcement learning attacks (RL-based Methods)

Prompt generation is modeled as an interactive environment, allowing the attack strategy to adapt dynamically based on system feedback.

• Policy interaction process: The attack model interacts directly with the defended system. For each test sample, it first sends an initial candidate and obtains the model response; the attack model then reflects on the response and score and proposes a new candidate. This process makes 5 attempts per round, and a single session measures attack success by the highest score among the 5 attempts.
• Policy update algorithm: For each sample, 32 independent sessions are executed, and the GRPO (Group Relative Preference Optimization) algorithm is used to update the attack policy model weights across sessions.
• Scoring design: For RPO and Circuit Breakers, the scoring function is the model's perplexity on the target malicious token plus the HarmBench classifier safety score; for StruQ, only the perplexity of the target string is used; for Data Sentinel, the sum of the target task string perplexity and 1.5 times the auxiliary model secret key perplexity is used. In experiments, a closed-source model without safety alignment is used as the attack base model.

Search-based methods

Adversarial prompt optimization is modeled as a discrete combinatorial search problem, implemented with a modular architecture based on the OpenEvolve framework.

• Controller: Uses the MAP Elites algorithm, dividing grid storage cells based on two features of candidate prompts—length and diversity—with each cell maintaining the best-performing sample (Elite), and maintaining a candidate pool across islands, randomly selecting historical samples for mutation.
• Mutator: Calls the large model LmMutator (using Gemini-2.5 Pro with a maximum thinking budget set) to perform mutation; the system prompt includes role setting, attack task (target tool invocation), and format rules, while the user message passes 8 historical candidates sampled by the controller along with their scores.
• Scorer: Combines the target system's qualitative text output with a 1-to-10 numerical score from an independent judge large model to generate feedback for each candidate and return it to the database.

Gradient-based and human red team implementations

• Gradient attack: Follows the GCG method, computing the gradient of the loss with respect to token embeddings in embedding space and projecting back to discrete tokens; the paper applies this method only to the RPO defense, using the model's negative loss (-loss) as the scoring mechanism.
• Human red team: Organized an online competition with more than 500 participants and a total prize of $20,000, setting up 40 challenge tasks based on AgentDojo; participants entered prompts through a web interface and viewed agent tool invocation traces in real time, with the environment automatically determining whether the injection task was accomplished, and scoring based on 5000 minus the number of submitted tokens to encourage short prompts.

Experimental Setup

• Models tested: Includes Gemini-2.5 Pro, GPT-5 Mini, Llama-3.3 70B, MetaSecAlign 70B, and for human red teaming: gemini-2.5-pro-thinking, gemini-2.5-flash, gemini-2.5-flash-thinking, grok-4, command-a-reasoning, gpt-5-medium, gpt-5-low, gpt-5-mini-medium, gpt-5-mini-low, gemini-2.5.
• Evaluation benchmarks: Jailbreak evaluation uses HarmBench; prompt injection evaluation uses AgentDojo (including Slack, Travel, Workspace suites with 80 samples total, skipping the Banking suite), OpenPromptInject, and the Davinci task based on Alpaca (davinci_003_outputs.json).
• Comparison baselines: Static attack prompts reported in the original papers of each defense (e.g., AgentDojo's static important instruction templates) or non-adapted weak optimization attacks.
• Metric definitions: Attack Success Rate (ASR, the proportion of samples achieving injection or jailbreak goals), benign Utility (the proportion of normal tasks completed without attacks), and median number of queries to first success (Median Num. Queries).
• Review and adjudication: Jailbreak determination uses HarmBench classifier scoring; prompt injection in the AgentDojo environment is determined by whether the environment judges the injected task tool call as successfully executed (human appeals are reviewed by human judges); data redirection tasks are judged by whether the target output matches the malicious category.
• Sample size and repetition budget: AgentDojo automated search evaluation uses 80 samples with a maximum query budget of 800 per sample; RL attacks run 32 independent sessions per sample, each iterating 5 rounds; human red teaming involves over 500 participants submitting multiple rounds for 40 challenges.

Main Results

The table below shows the search attack ASR performance of 4 base models under different defenses on the AgentDojo benchmark (data from Table 7):

Swipe horizontally to read the table

Note: Data from Table 7 on AgentDojo search attack ASR results. Due to table column limits, other defenses in Table 7 (Prompt Sandwiching at 69%–100%, Protect AI at 81%–94%, Model Armor at 56%–90%) are not listed here.

• Defenses weaken under adaptive attacks: The authors state that most model-defense combinations were broken under adaptive search attacks, with ASR exceeding 80% for most combinations, except GPT-5 Mini with some defenses and PIGuard against stronger models (Table 7).
• Query budget and convergence characteristics: The authors report that in most scenarios, search attacks found effective triggers within 100 to 200 queries, not exhausting the 800-query limit, with median queries between 13 and 110 (Table 7).
• Contrast between static and adaptive evaluation: The authors state that ASR under static attacks was generally low across models (e.g., GPT-5 Mini had 0% static ASR under several defenses), but adaptive attacks raised success rates to 47%–100%, indicating that static test sets give a misleading sense of security (Table 7).

Human Red Team Evaluation and Query Count Comparison

• What was tested: Organized a red team attack-defense competition with over 500 participants in the AgentDojo environment, targeting 5 defenses and 10 models, and compared query efficiency between human red team and automated search attacks across 29 overlapping scenarios (Figure 6).
• Experimental results: In the selected 29 overlapping scenarios, the collective human red team success rate reached 100%, with the best-performing participant needing only 50 queries to break a scenario; in contrast, automated search attacks without random restarts achieved 69% ASR under an 800-query budget (Figure 6, Section 6).
• Authors' interpretation: The authors state that human expert intuition and targeted exploration remain an important complement to automated tools, being more effective in dynamic interactive environments; however, automated search still demonstrates scalable bypass capability without human intervention.

Adaptive Bypass of Secret Knowledge and State Detection Defenses

• What was tested: Against detection mechanisms relying on unknown secret assumptions, Data Sentinel (honeypot key repetition task) and MELON (dual-run comparison of tool calls), using RL attacks and mutation search attacks based on defense knowledge, respectively.
• Experimental results: On Data Sentinel, RL attacks achieved over 80% target redirection accuracy on both sentiment analysis and spam detection tasks (Section 5.4); on MELON, search attacks without explicit defense knowledge achieved 76% ASR on Llama-3.3-70B, while with an adaptive mutator given the full defense implementation, ASR reached 95% (Section 5.4).
• Authors' interpretation: The authors state that defenses relying on hidden information assumptions are not robust; attackers can bypass detection by redefining task instructions without changing output format. When attackers know the detection logic, they can add conditional branch instructions in the payload to separate the verification run from the real run.

Utility Degradation and False Positive Costs of Filtering Model Detectors

• What was tested: Evaluated the interception performance of external standalone detectors such as Protect AI, PromptGuard, PIGuard, and Model Armor under adaptive search attacks and their impact on benign task Utility.
• Experimental results: Search attacks achieved over 90% ASR against Protect AI, PromptGuard, and Model Armor (with Gemini-2.5 Pro as base), and 71% against PIGuard (Table 7); additionally, filtering defenses reduced benign utility, e.g., on Gemini-2.5 Pro, Utility with Protect AI and PIGuard dropped to 39.18%, compared to 74.23% without defense (Table 7).
• Authors' interpretation: The authors state that external classifiers are themselves susceptible to adversarial perturbations and cannot effectively distinguish redirection triggers without obvious malicious intent; forcibly increasing sensitivity raises false positive rates on benign tasks and harms usability.

Other Ablations and Analyses

• Performance of the RPO jailbreak defense under different attack variants: RL attack ASR 98%, gradient attack ASR 96% (Section 5.1).
• Performance of the MELON defense under different attacker knowledge levels: black-box search without defense knowledge ASR 76%, adaptive search with full knowledge of the defense implementation ASR 95% (Section 5.4).
• Median query counts for search attacks on different models (no defense): Gemini-2.5 Pro 13, Llama-3.3 70B 14, MetaSecAlign 70B 37, GPT-5 Mini 58 (Table 7).
• Individual human red team performance variation: the best participant completing all 29 scenarios had an ASR of 75%, while the other two participants completing all scenarios had ASRs below the automated search's 69% (Section F.2).
• Evolutionary algorithm structure ablation: the authors compared the MAP Elites island model with a single-database evolutionary algorithm, ultimately adopting the better-performing MAP Elites island model without tuning subsequent hyperparameters (Section D).

Limitations and future directions noted by the authors

• Reducing the computational cost of strong attacks: In Section 3, the authors note that the extent to which future research can achieve the strong adaptive attacks demonstrated in this paper with fewer computational resources is a direction worth exploring.
• Developing more stable adaptive evaluation pipelines: In Section 6, Lesson #2, the authors note that current reinforcement learning and search attacks are still less stable than image adversarial attacks (such as PGD), and call on the community to continue developing more robust adaptive evaluation algorithms.
• Improving parallelization mechanisms for search algorithms: In Section G.1, the authors note that current search attacks tend to plateau after 100 to 200 queries, and speculate that search algorithms that better exploit parallelism may benefit more than simply increasing the number of sequential iterations.
• Standardizing control variables in manual red team evaluations: In Section F.2, the authors note that human attackers invest uneven effort across different scenarios, and if all human participants were required to use the same number of attempts in each scenario, the comparison results would be more referential.
• Benchmark design for plan-then-execute defenses: In Section G.1, the authors note that plan-then-execute defenses cause existing attacks to inevitably fail because control flow is independent of data, and future evaluation requires constructing scenarios where control flow does not fully depend on data through subsampling.
• Security flaws in automated model scorers: In Section 6, Lesson #4, the authors note that the classifier models used for automated safety evaluation are themselves machine learning models and are susceptible to adversarial examples and reward hacking.

Experimental coverage

• The tested defense mechanisms cover 4 categories totaling 12 defenses: prompt engineering, adversarial training, external filtering models, and secret knowledge; plan-then-execute defenses such as Tool Filter, CaMeL, and IPIGuard were skipped (Section 5, Section G.1).
• Prompt injection evaluation covers AgentDojo (including Slack, Travel, and Workspace suites totaling 80 samples; the Banking suite was not tested), OpenPromptInject, and Alpaca; jailbreak evaluation covers HarmBench (Section B.1).
• The maximum query budget per sample for automated search attacks was set to 800, and the experiments ran only a single attempt without multiple random restarts (Section 6, Section F.2, Table 7).
• The paper does not report the specific model name or version of the closed-source base model without safety alignment used for reinforcement learning attacks (Section A.2).
• The mutator for search attacks was fixed to Gemini-2.5 Pro with a maximum thinking budget; the paper does not test the impact of other mutation models on attack effectiveness (Section D).

• One-sentence positioning: This is an empirical study systematically examining the true robustness of jailbreak and prompt injection defenses for large models, proposing a general adaptive attack framework and evaluation paradigm.
• Problem to solve: Current academic and industrial evaluations of LLM defenses commonly rely on static sets of malicious prompts or low-compute generic non-adapted algorithms, so evaluation results fail to reflect defense effectiveness against real adaptive attackers.
• Method highlights:
  1. Formalized the PSSU adaptive attack loop consisting of propose, score, select, and update.
  2. Implemented four attack variants: gradient-based coordinate descent (GCG), reinforcement learning-based policy optimization (GRPO), combinatorial search based on MAP Elites with LLM mutations, and bonus-incentivized crowdsourced human red teaming.
• Key experimental findings:
  1. Adaptive attacks broke the near-zero success rates reported in the original papers across 12 defense mechanisms, with most defenses achieving ASR above 90% (Figure 1).
  2. On the AgentDojo benchmark, automated search attacks achieved 95% to 100% ASR on Gemini-2.5 Pro and Llama-3.3 70B with prompt-based defenses (Spotlighting, Prompt Sandwiching) (Table 7).
  3. Against external filtering models (Protect AI, PromptGuard, Model Armor), search attacks generally exceeded 90% ASR, and the filtering models reduced benign task utility from 74.23% to 39.18% (Table 7).
  4. Against the MELON defense that relies on secret knowledge, adaptive search with knowledge of the defense logic achieved 95% ASR; in human red team tests, human attackers achieved 100% success across all 29 overlapping scenarios (Section 5.4, Figure 6).
• Authors' conclusions and implications: The authors argue that reporting near-zero ASR on static benchmarks does not indicate defense robustness; defense evaluation should, as in computer security, assume attackers know the defense mechanism and invest sufficient compute for adaptive bypass. They also state that human red teaming remains an important complement to automated evaluation at the current stage, and future defense design should aim to be simple, interpretable, and amenable to human audit.