Pair-Aware Optimization: Coordinating Exploration and Refinement for LLM Agent Jailbreaks

1. 01What problem does this paper attempt to solve?

   How ZO and FO should be ordered under stochastic decoding remains unclear, and the authors use Pair-aware to coordinate prefix and suffix.

   Under stochastic decoding, how to organize ZO and FO to optimize an adversarial prefix–suffix and improve targeted jailbreak success rate.

   • Scenario: the authors consider that LLM agents can be induced by adversarial inputs to produce attacker-specified malicious behaviors. The same input under stochastic decoding can produce different reasoning trajectories, tool calls, and final actions, which a single greedy output cannot cover.
   • Existing approaches: the authors state that existing optimization-based attacks mainly use FO gradient information or ZO function-value feedback; how the two complement each other and which comes first is still relatively little studied. The hybrid methods in the introduction combine search and gradient adaptation, and the authors state that organizing these two paradigms directly on adversarial inputs is still relatively little examined.
   • Observation: the intermediate solution of the first stage determines where the second stage continues, and the order changes the trajectory. The authors also point out that the strongest ZO candidate before refinement is not necessarily still the best after refinement.
   • This paper proposes: comparing pure ZO, pure FO, ZO→FO, and FO→ZO, using ZO to explore the prefix and FO to refine the suffix, and using Pair-aware to select the complete pair after conditional refinement. Optimization and evaluation use the same decoding distribution.
   • Threat model:
     • Knowledge: white-box. Can access the victim model's output probabilities and input gradients, but cannot modify parameters, system prompt, or tools.
     • Capability: only optimizes the specified adversarial span, written as an adjacent prefix P and suffix S, with the rest of the context fixed. ZO searches discrete candidates via objective feedback, and FO makes local token updates via gradients.
     • Goal: maximize the expected attack reward under stochastic decoding; when the reward is a binary indicator, this expectation is the probability of the target behavior occurring.
     • Target: the victim model is an LLM agent. The paper studies both malicious-instruction and malicious-environment settings.

   Collapse this answer

2. 02What related research exists?

   Related work divides into gradient, gradient-free, and hybrid optimization; the authors focus on the order on adversarial inputs.

   The authors discuss related work by gradient, gradient-free, and hybrid optimization, and position this paper in the combination and ordering of ZO/FO on adversarial inputs.

   Gradient-Based Jailbreaks

   • Representation and objectives: Wang et al. (2024a) and Lin et al. (2024) optimize in embedding or representation space; Liu et al. (2024a) and Wang and Qi (2025) do staged suffix refinement and augmented objectives; LeVi et al. (2025) analyze initialization and the compliance direction.
   • Web agents: WebInject (Wang et al., 2025) extends gradient attacks to web agents using a differentiable rendering proxy.
   • Authors' distinction: the authors state these methods mainly focus on first-order optimization, whereas this paper studies its combination and ordering with ZO.

   Gradient-Free Jailbreaks

   • Search and obfuscation: Wu et al. (2025) and Huang et al. (2025) use tree search and feedback-driven rewriting; Chen et al. (2025) and Zhang et al. (2025b) use wordplay and semantic obfuscation.
   • Prompt transformations: work by Hsu, Dong, Zou, Zheng, Joo et al. (2025) covers multi-prompt, task masking, structured queries, defensive framing, and symbolic encoding.
   • CEM and indirect injection: Chang et al. (2026) use CEM to optimize retrieval trigger prefixes with black-box similarity feedback. The authors state most of these methods are independent of first-order optimization, unlike this paper's study of FO–ZO combination and ordering.

   Hybrid Optimization

   • Allocation in training: Li et al. (2025b) and Shirkavand et al. (2026) combine gradient estimates or assign different optimizers to model components.
   • Heterogeneous updates: Yang and Aamodt (2026) and Zhang et al. (2026) use FO to guide ZO updates, or alternate or jointly use heterogeneous rules. The authors state these works demonstrate FO–ZO complementarity in model training and do not study the combination and ordering on adversarial inputs when jailbreaking LLM agents.
   • Search plus gradients: the introduction also cites Paulus et al. (2024), Sun et al. (2025), and Basani and Zhang (2026). The authors state gradient-free exploration and gradient refinement can be complementary, but how to organize them directly on adversarial inputs remains less studied.

   Baselines and Benchmarks

   • Baselines: GCG does gradient-guided coordinate updates; REINFORCE-GCG replaces fixed target likelihood with an adaptive semantic reward; UDora's Sequential and Joint construct proxy objectives from intermediate trajectories and then do gradient-based adversarial optimization.
   • Benchmarks: AgentHarm, InjecAgent, WebShop, with the experimental protocol following UDora.

   The authors position this paper as: not presupposing an order, comparing four organizations, with ZO exploration and FO refinement as the basic framework, and Pair-aware coordination of the two stages; evaluation is on stochastic generation rather than single greedy outputs.

   Collapse this answer

3. 03How does the paper address this problem?

   CEM explores the prefix, FO refines the suffix, and Pair-aware selects pairs after refinement by success first and then by loss.

   Pair-aware optimization formulates jailbreak as discrete optimization over an adjacent prefix and suffix: ZO explores the prefix, FO refines the suffix, and commitment is deferred until the refined whole pair's result is seen. Figure 1 uses blue and orange to distinguish prefix and suffix; Figure 2 depicts UDora as deterministically choosing insertion positions, and Pair-aware as conditional refinement followed by selecting the pair by success first, then loss.

   Problem setting and surrogate objective

   The victim agent has parameters θ and decoding configuration R. Given the original input x and an adversarial pair (P, S), trajectories are sampled from that conditional distribution. The authors aim to maximize the expected attack reward

   J(P, S)=𝔼_{Z∼πθ,R(·|x,P,S)}[r(Z)]

   When r is a binary indicator of whether the target behavior is achieved, J is the probability of that behavior under stochastic decoding. Directly optimizing J is infeasible. ZO ranks discrete candidates by the average teacher-forced cross-entropy of the target sequence; FO takes gradients of the target loss from rollout conditions, then filters with another surrogate loss. Optimization and evaluation use the same stochastic decoding distribution.

   How ZO and FO produce candidates

   • ZO (CEM): editable positions independently sample tokens from the current distribution, with the rest of the input fixed. Legal candidates and the current string form a pool, and the one with lowest cross-entropy is selected; low-loss candidates then update the sampling distribution. Figure 1 states that this path does not use model gradients.
   • FO: does not update the victim model's parameters. A surrogate context is constructed from the current rollout, the gradient of the target loss is taken with respect to a one-hot relaxed representation of the attack tokens, local replacements are generated, and the one with lowest filtering loss is selected. The context is fixed when computing gradients and comparing candidates.
   • Division of the two losses: constructing candidates uses the rollout-conditioned target cross-entropy; ranking discrete candidates uses the UDora filtering loss. For one target span,

   L_U(s′_i)=−(m+p̄)/(n+1)

   m is the number of tokens from the start of the span that consecutively match the teacher-forced argmax; p̄ is the average probability of the target tokens up to the first mismatch (or over the whole span if all match). For multiple spans, the span losses are summed.

   Four organizations

   All hybrid settings update the prefix first, then the suffix, only reversing the optimizer order. The other segment is fixed during that stage.

   • Pure ZO / Pure FO: the entire adversarial string uses only ZO or only FO.
   • FO→ZO: with the suffix fixed, FO updates the prefix, then ZO optimizes the suffix under the new prefix.
   • ZO→FO: with the suffix fixed, ZO updates the prefix, then FO refines the suffix under the selected prefix. The authors treat this as the basic organization and test it with ablations, without presupposing that one order is universally better.

   How Pair-aware coordinates

   Algorithm 1 has three steps per round, stops if there is already a success, and runs at most T rounds in the outer loop. Lengths, steps, and sampling in the experiments are given in the next question.

   1. Prefix exploration: with the current suffix fixed, CEM samples prefixes and retains at most K low-loss prefixes by cross-entropy paired with that suffix. These scores only determine who enters refinement.
   2. Conditional refinement: each retained prefix starts from the same current suffix and independently performs FO. Gradient proposals generate full suffix candidates, and generation and selection are repeated by filtering loss; the context is fixed within one comparison and refreshed between local steps. Then one stochastic rollout is performed, recording the success flag and the outer objective cross-entropy.
   3. Pair selection: the pool contains the current pair and all refined pairs. Attack success is prioritized first, then lower target cross-entropy. The prefix and suffix are replaced together only if the new pair succeeds or has lower loss; otherwise the current pair is kept. The inner filtering loss does not participate in this outer ranking.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On two 8B models and three benchmarks, Pair-aware's average ASR is higher than the listed optimization baselines.

   Experimental setup

   • Models: two instruction-tuned models, Llama-3.1-8B-Instruct and Qwen3-8B; the latter has thinking mode disabled. Both serve as tool-using agents. Optimization and evaluation share the same model configuration.
   • Benchmarks: AgentHarm selects 44 harmful prompts from the public test set, with four configurations being instruction detail level crossed with whether there is a first tool-call hint, belonging to malicious instructions. InjecAgent contains 17 user tools and 62 attacker tools, with 50 cases each for Direct Harm and Data Stealing, looking only at the first malicious action; the adversarial string is appended after the malicious instruction inserted into the first tool observation. WebShop replaces retrieval results with attacker-specified targets, with four targets Price, Attribute, Category, All Mismatch, each target selecting three instructions from each of five product categories, and the adversarial string inserted into the target product title. The protocol follows UDora.
   • Baselines: GCG, REINFORCE-GCG, UDora (Sequential) and UDora (Joint). The authors state that the latter two set num_locations to 2 and 4, corresponding to their stronger greedy position-selection configuration.
   • Metrics: ASR is the number of successful cases divided by the total number of cases, multiplied by 100%. The first two benchmarks count success as executing the specified malicious tool call or target behavior; WebShop counts success as the final selection of the specified target ASIN. The paper does not state an LLM judge model.
   • Optimization values: agents use ReAct prompting. For AgentHarm, search width, gradient top-k, and maximum steps are 256, 64, 1000; for the other two benchmarks they are 128, 32, 500. The adversarial string is initialized as 25 repeated x; the hybrid method is split into a 10-token prefix (ZO) and a 15-token suffix (FO). The optimization target is the adversarial tool name or target ASIN. Pair-aware refines at most four prefixes per round, sharing the step budget.
   • Decoding: all methods use stochastic decoding in both optimization and evaluation, temperature 0.7, top-p 0.9. The paper does not report the number of repetitions.

   Main results

   The table below gives the average ASR from Tables 1–3. InjecAgent keeps integer percentages as in the original source.

   Swipe horizontally to read the table

   Model and benchmark	GCG	UDora (Sequential)	UDora (Joint)	REINFORCE-GCG	Pair-aware
   Llama-3.1-8B-Instruct, AgentHarm	37.50%	50.57%	44.89%	75.00%	89.20%
   Qwen3-8B, AgentHarm	76.70%	80.68%	81.82%	85.23%	93.18%
   Llama-3.1-8B-Instruct, InjecAgent	31%	45%	48%	46%	100%
   Qwen3-8B, InjecAgent	5%	3%	5%	8%	34%
   Llama-3.1-8B-Instruct, WebShop	0.00%	8.33%	8.33%	1.67%	26.67%
   Qwen3-8B, WebShop	5.00%	6.67%	6.67%	8.33%	28.33%

   • AgentHarm: the authors state that Pair-aware is higher than the baselines on all four configurations for Llama-3.1-8B-Instruct, and higher on three for Qwen3-8B. On Detailed+Hint, Pair-aware is 97.73%, lower than REINFORCE-GCG's 100.00% (Table 1). The authors report average gaps relative to this baseline of 14.20% and 7.95%.
   • InjecAgent: the authors state it is higher than each baseline on both categories and both models. For Qwen3-8B, Direct Harm is 32% and Data Stealing is 36%. The authors state it is on average 52 percentage points higher than UDora (Joint) and 26 percentage points higher than REINFORCE-GCG (main text, relative to Table 2).
   • WebShop: the authors state it is higher than each baseline on three categories for Llama-3.1-8B-Instruct and four categories for Qwen3-8B, with Attribute Mismatch at 60.00% and 53.33% respectively. The authors report being on average 18.34 and 20.00 percentage points higher than the strongest baseline. In Table 3, on All Mismatch for Llama, Pair-aware is 6.67% and the baseline in that column is 0.00%; this cell is also higher than the baseline, which is not fully consistent with the "three categories" summary.

   Optimization order ablation

   • Setup: Section 5 keeps Pair-aware's conditional refinement and pair selection, fixes updating the prefix first then the suffix, and only changes the optimizer used for the two segments, yielding pure ZO, pure FO, ZO→FO, FO→ZO.
   • AgentHarm (Table 4): ZO→FO has the highest average. The authors state it is 28.97 and 7.38 percentage points higher than FO→ZO. Pure ZO has the lowest average, at 50.00% and 78.41%. FO→ZO versus pure FO is 60.23% versus 59.09% for Llama-3.1-8B-Instruct, and 85.80% versus 86.36% for Qwen3-8B. On Detailed+Hint for Qwen3-8B, FO and ZO→FO are both 97.73%.
   • InjecAgent and explanation (Table 5): ZO→FO is still highest; the authors state it is on average 45 and 31 percentage points higher than FO→ZO. For Qwen3-8B, the average ASR of pure ZO, pure FO, and FO→ZO is 2%, 4%, 3%. The authors argue the order itself matters: a prefix that performs well under the current suffix may become worse after refinement; FO-only may search along similar gradient trajectories, ZO-only lacks rollout-conditioned gradient refinement, and FO→ZO performs distribution exploration only after gradient optimization, so new candidates may not receive further refinement.

   Appendix cases

   • Scope: Appendix A uses one example from each of the three benchmarks to show candidate filtering, conditional refinement, and pair selection, not as aggregate metrics. The user requests and tool returns therein are not restated.
   • Success priority: Figure 3 marks the losses of four refined pairs; Pair 3, which hits the target, has 4.34375, higher than the non-hitting Pair 1 (3.56250), and is still selected.
   • The other two examples: in Figure 4 the second pair elicits the specified tool call; in Figure 5 the fourth pair elicits the target click. The figures state that if unsuccessful, filtering and refinement are repeated.

   Other ablations and analyses

   • Apart from Tables 4 and 5, the paper does not report sweeps over K, temperature, prefix/suffix length, or steps.
   • Table 5: for Qwen3-8B, pure ZO, Direct Harm, the ASR is 0%.
   • Table 3: on All Mismatch, Pair-aware is 6.67% for both models.
   • The paper does not report the number of queries, runtime values, or variance across repeated experiments.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that multi-prefix refinement increases overhead, and suggest adaptively allocating candidates and pruning.

   Limitations and Future Directions Noted by the Authors

   • Computational cost: refining multiple prefixes per condition and evaluating the resulting prefix–suffix pairs incurs extra computation (Limitations).
   • Runtime: compared with single-prefix sequential optimization, this pairwise evaluation may increase runtime, especially when many candidates are retained (Limitations).
   • Budget trade-off: under a fixed compute budget, there is a trade-off between prefix coverage and suffix refinement depth (Limitations).
   • Algorithmic follow-up: the authors propose studying adaptive candidate allocation and selective branch pruning to reduce computational cost while retaining Pair-aware coordination (Limitations).
   • Evaluation and defense: in the Conclusion, the authors call for more complete safety evaluation and adaptive defenses that can handle different generation trajectories and changing adversarial optimization strategies.

   Experimental Coverage

   • Models tested are Llama-3.1-8B-Instruct and Qwen3-8B, 2 in total; Qwen3-8B has thinking mode disabled (Section 4.2).
   • AgentHarm is the public test set of 44 items under four configurations; InjecAgent has 50 examples in each of two categories; WebShop has four goals, each selecting three instructions from each of five product categories (Section 4.1).
   • Controls are GCG, REINFORCE-GCG, UDora (Sequential), and UDora (Joint). Success is determined by tool call, target behavior, or target ASIN; the paper does not state a judge model (Section 4.2).
   • The attack setting is white-box; optimization and evaluation use the same stochastic decoding, temperature 0.7, top-p 0.9. Optimization-mode ablations are AgentHarm in Table 4 and InjecAgent in Table 5.
   • The paper does not report number of repetitions, number of queries, or specific runtimes; WebShop does not appear in the optimization-mode ablation tables.

   Collapse this answer

6. 06Summarize the main content of the paper

   Pair-aware jailbreaks the agent with ZO→FO, and the authors state its average ASR is higher than the listed optimization baselines.

   Pair-aware optimization, under white-box and stochastic decoding, formulates LLM agent jailbreaking as optimizing over adjacent adversarial prefixes and suffixes.

   • Problem: how to order FO gradient refinement and ZO function-value exploration, and how to avoid committing to a single prefix before suffix refinement. The same input elicits different trajectories, so the authors have optimization and evaluation share the decoding distribution.
   • Method: CEM explores multiple prefixes with the suffix fixed; FO proposes suffix replacements using gradients of the rollout-conditioned objective loss, and ranks them with the UDora screening loss. Pairwise selection first checks whether the stochastic rollout achieves the target behavior, then the teacher-forced target cross-entropy; prefix and suffix are accepted or retained together.
   • Results: on Llama-3.1-8B-Instruct, Pair-aware average ASR is 89.20% on AgentHarm (Table 1), 100% on InjecAgent (Table 2), and 26.67% on WebShop (Table 3). On Qwen3-8B they are 93.18%, 34%, and 28.33%, respectively. The authors state these averages are higher than GCG, REINFORCE-GCG, and both UDora variants.
   • Exceptions and ordering: on Qwen3-8B AgentHarm Detailed+Hint, Pair-aware is 97.73%, below REINFORCE-GCG's 100.00%. In the ablation, ZO→FO averages higher than pure ZO, pure FO, and FO→ZO (Table 4, Table 5).
   • Authors' conclusions: the authors argue agent vulnerability is related both to the stochastic behavior distribution and to how adversarial optimization is organized, and call for more complete safety evaluation and adaptive defenses. Multi-prefix refinement increases computation, and under a fixed budget one must trade off prefix coverage against suffix depth.

   Collapse this answer

Related papers