Black-box optimization of LLM outputs by asking for directions
Black-box Optimization of LLM Outputs by Asking for Directions
The authors propose having the target LLM compare two candidate inputs in text to guide black-box optimization, reporting high ASR on jailbreaking, prompt injection, and visual adversarial examples.
The attacker has only query access to the target model (text-only output): no access to parameters, gradients, logits, or token log-probabilities, and can only observe text output; no surrogate or auxiliary models are used.
- Black-box attacks on LLMs typically require logits or confidence scores, or rely on surrogate models; under an API that returns only text, optimization signals are lacking. The authors ask: can the target model itself express confidence in text to guide adversarial optimization?
- Reformulate optimization as a series of binary comparisons: at each iteration, randomly perturb the current input, ask the model which candidate is closer to the attack goal, and accept if it is selected (hill-climbing). The authors state that binary comparison is better calibrated than directly asking for absolute confidence, and less likely to trigger refusals.
- Evaluated on three task types: ImageNet adversarial examples, AgentDojo prompt injection, and AdvBench jailbreak. For jailbreak, 5 of 6 models reach ASR 98%–100%; prompt injection ASR rises from 23.2%–42.1% to 55.4%–87.5%; the authors state that larger, stronger models are more vulnerable.
- Failure modes noted by the authors: weaker models cannot compare candidates; comparisons have false positives/false negatives; strongly aligned models refuse to compare (e.g., GPT-4o mini during jailbreaking). The authors did not do extensive hyperparameter tuning, nor explore other perturbation methods; defenses are only discussed, not empirically tested.
- Evaluated models
- Qwen2.5-VL-3B-InstructQwen2.5-VL-7B-InstructQwen2.5-VL-72B-InstructLlama-3.2-11B-VisionLlama-3.2-90B-VisionGPT-4o miniGPT-5 miniClaude Haiku 3.5Claude Sonnet 3.7Claude Opus 4Llama-3.1-70B-InstructGPT-4oLlama-3.1-8BGPT-4 TurboGPT-4.1 miniClaude Sonnet 3.5Claude Sonnet 4
- Benchmarks
- ImageNetAgentDojoAdvBench
- Metrics
- ASRMean queries
Researchers propose a black-box attack optimization method that relies only on text outputs, obtaining optimization signals by having the model compare two candidate inputs, and apply the method to vision LLM adversarial examples, jailbreaks and prompt injection. The authors find that directly asking for absolute confidence collapses to a few values such as 0, 5, 95, whereas binary comparisons are better calibrated; on 56 AgentDojo prompt injection samples, the success rate rises from 23.2% to 42.1% to 55.4% to 87.5%, and jailbreaks reach 100% on Llama-3.1, GPT-4 Turbo and Claude Sonnet 3.5. The authors also claim that larger, more capable models have better-calibrated comparative confidence and are therefore more susceptible to such attacks.
Why it was selectedThe paper shows that jailbreaks and prompt injection can be optimized using only text outputs, and provides data showing that more capable models are more susceptible to such comparative queries.
Paper analysis
What problem does this paper attempt to solve?
Under a black-box API that returns only text, how to perform adversarial optimization on an LLM without a surrogate model.
Core question: in a black-box setting where the target LLM exposes only text output (text-only), how can one obtain a sufficiently good optimization signal to generate adversarial inputs.
- Scenario and importance: LLMs are deployed in chat, code assistant, agent and other systems, and the attack surface includes jailbreaking, prompt injection, and visual adversarial examples; the authors consider the text-only setting increasingly common in practice (e.g., Anthropic's Claude API deliberately exposes less information).
- Shortcomings of existing methods: white-box attacks require gradients; with logits/confidence, gradient-free optimization can be used; under text-only, existing attacks rely on surrogate signals—optimizing on a local surrogate and then transferring, or using an external reward model to guide—and the authors consider these surrogate signals potentially inconsistent with the target model's behavior.
- Core observation: directly asking the model for absolute confidence scores works poorly (values collapse to a few values such as 0, 5, 95, and it may refuse); but the authors find that the model is better calibrated when comparing which of two candidate inputs is closer to the attack goal, and such queries are usually considered harmless.
- The paper proposes: a general query-based black-box optimization method—binary-comparison-guided hill-climbing—applied to three scenarios: adversarial examples, jailbreaking, and prompt injection.
- Threat model:
- Goal: per scenario, cause misclassification (untargeted/targeted), make the agent execute a malicious function, or induce harmful responses.
- Knowledge: black-box, query access only; text-only output, no visibility into parameters, gradients, logits, token log-probabilities.
- Capability: can submit arbitrary prompts and observe text output; no surrogate or auxiliary models are used.
- Victim system: deployed models (vision-LLM or LLM) accessed via API or web interface.
What related research exists?
Related work falls into two categories: black-box attacks (manual prompting, LLM-guided, transfer, query-based) and LLM confidence calibration.
Black-box attacks—manually designed adversarial prompts
- Zvi (2022), Wei et al. (2023a;b), and others found various jailbreak techniques; Willison (2022), Goodside (2022), Greshake et al. (2023), and others proposed prompt injection. Strategies include role-playing (Shen et al., 2024), instruction override (Perez & Ribeiro, 2022), and others. The paper notes that such attacks have high success rates but are labor-intensive and do not follow explicit optimization.
Black-box attacks—LLM-guided and transfer
- PAIR (Chao et al., 2025) uses an attacker LLM to iteratively rewrite prompts; Mehrotra et al. (2024), Sitawarin et al. (2024), and others use auxiliary LLMs for red-teaming; Beutel et al. (2024) trains an attack model with multi-step reinforcement learning. The paper notes that these methods use auxiliary models for ad-hoc optimization, whereas this paper optimizes directly on the target model.
- Transfer attacks: Szegedy et al. (2013) and Papernot et al. (2016) laid the foundation; Li et al. (2025), Hu et al. (2025), and Jia et al. (2025) target vision-LLMs, while Zou et al. (2023), Andriushchenko et al. (2024), and others target jailbreaks. The authors state that this paper is the first method to directly optimize adversarial examples on black-box vision-LLMs without relying on transferability.
Query-based attacks
- Chen et al. (2017), Brendel et al. (2017), and Cheng et al. (2018) are well studied in the traditional adversarial example domain; the paper notes that successful query-based attacks on LLMs are limited and depend on the availability of confidence scores (Andriushchenko et al., 2024; Hayase et al., 2024).
Elicitation and calibration of LLM confidence
- Jiang et al. (2020), Xiong et al. (2024), Yang et al. (2024), and others show that LLMs can report and calibrate their own confidence; Kadavath et al. (2022) state that self-reported confidence is sometimes better calibrated than token log-probability; self-consistency (Wang et al., 2024) and SelfCheckGPT (Manakul et al., 2023) provide implicit confidence signals.
Baselines and benchmarks: The baselines are pre-optimization ASR, stronger attacks with access to log-probabilities (the method of Andriushchenko et al., 2024, implemented by the authors themselves for adversarial examples and prompt injection), and transfer attacks (transferable adversarial examples generated with three CLIP models of different sizes, following Li et al., 2025). The benchmarks are ImageNet, AgentDojo (Debenedetti et al., 2024), and AdvBench (Zou et al., 2023, using the 50 items curated by Chao et al., 2025). The authors' positioning: use the model's own introspective capability in place of external surrogate signals as the black-box optimization signal.
How does the paper address this problem?
Model the attack as constrained optimization, using the target model's binary comparison answers for hill-climbing search.
Overall idea: formulate the black-box attack as a constrained optimization problem; since gradients or scores are unavailable, instead repeatedly submit binary comparison queries to the target model (which of two candidates is closer to the attack goal), and use the answers to drive hill-climbing.
Problem formulation
- Goal: find an adversarial input xadv such that M(xadv) ∈ G and xadv ∈ C(x), where G is the attack goal and C(x) is the constraint set.
- Constraints are defined per domain: for images, an ℓ∞ ball (perturbation ≤ ϵ); for text, a suffix s appended after the original prompt, with length |s| ≤ N.
Optimization signal: binary comparison
- Directly asking for absolute confidence (e.g., "score from 1–100") is shown by the authors to fail: in Figure 3(a), Qwen2.5-VL-72B-Instruct's expressed confidence takes only a few values such as 0, 5, 95, and cannot distinguish subtle improvements; the model may also refuse to answer.
- Instead, ask "which of the two inputs is closer to the attack goal G". Figure 3(b) shows that on Qwen2.5-VL-72B-Instruct, the model's binary answers are on average calibrated with the logit gap: false positives occur only when the gap between the two candidates is small, false negatives are more numerous but only slow down optimization, and true positives guide in the correct direction. The authors also state that closed-source models are generally more "permissive" (more readily accept updates).
Algorithm procedure (Algorithm 1)
- Initialize xadv ← x; iterate at most T times.
- Randomly perturb xadv within constraint C to generate xnew.
- Submit a binary comparison query to M; if xnew is preferred (r=1), accept xadv ← xnew.
- Check whether M(xadv) ∈ G; if successful, return.
Perturbation strategies
- Adversarial examples: adopt Square Attack (Andriushchenko et al., 2020), adding ℓ∞ perturbation (bound ϵ) in a random square region of the image at each step, using binary comparison in place of the original method's confidence score to decide whether to keep the change.
- Prompt injection and jailbreaking: append a suffix of length N after the prompt; at each step, randomly select a contiguous segment of the suffix and replace it with tokens randomly sampled from the vocabulary; the authors state this is similar to the attack of Hayase et al. (2024), except that the latter uses the model's true confidence to guide it.
Success criteria
- Adversarial examples: for untargeted attacks, ask "Does this image contain a ⟨label⟩?"; a negative answer counts as success; for targeted attacks, ask "does it contain ⟨target_label⟩ but not ⟨source_label⟩"; an affirmative answer counts as success.
- Prompt injection: judged by whether the malicious function is executed (e.g., the model calls send_email).
- Jailbreaking: use Llama-3.1-8B and GPT-4o mini as judge models to determine whether the response is harmful. Comparison queries are designed according to model response style: GPT refusals often begin with "I'm sorry", Claude with "I do not", and other models indicate compliance with "Sure" (Appendix B); Figure 4 gives comparison query templates for jailbreaking and prompt injection, and Appendix D gives complete attack examples.
What experiments did the paper conduct?
17 models across three scenarios; jailbreak ASR ≥98% for 5/6 models, prompt injection rises to 55.4%–87.5%.
Experimental setup
- Models tested: adversarial examples on 10 vision-LLMs (Qwen2.5-VL-3B/7B/72B-Instruct, Llama-3.2-11B/90B-Vision, GPT-4o mini, GPT-5 mini, Claude Haiku 3.5, Claude Sonnet 3.7, Claude Opus 4); prompt injection on 5 (Llama-3.1-70B-Instruct, GPT-4o mini, GPT-4o, Claude Haiku 3.5, Claude Sonnet 3.7); jailbreaking on 6 (Llama-3.1-8B, Llama-3.1-70B, GPT-4 Turbo, GPT-4.1 mini, Claude Sonnet 3.5, Claude Sonnet 4).
- Datasets: 57 random ImageNet images (resized to 224×224, on which all target models have 100% clean accuracy); 56 samples adapted from AgentDojo (4 user task types, 14 injection attack types); 50 harmful requests from AdvBench curated by Chao et al. (2025).
- Baselines: ASR before optimization; attacks with log-probabilities (implemented by the authors following Andriushchenko et al., 2024); transfer attacks (generated from three CLIP models of different sizes, following Li et al., 2025).
- Metrics and judging: ASR; jailbreaking harmfulness judged by Llama-3.1-8B and GPT-4o mini as judges; prompt injection judged by whether the malicious function is executed; jailbreaking also reports average query count.
- Budget and parameters: adversarial examples query budget 1,000, ϵ=32/255; jailbreaking query budget 200. Prompt injection is an adaptive attack; GPT-4o mini and Claude Haiku 3.5 were given a system-prompt defense to avoid being compromised before optimization (Appendix B).
- Number of repetitions: not reported in the paper.
Main results
Jailbreaking (Table 3, AdvBench 50 items, budget 200 queries):
Swipe horizontally to read the table
Model Before optimization with logprobs Ours Average queries Llama-3.1-8B 8% 100% 100% 32.4 Llama-3.1-70B 0% 100% 100% 15.2 GPT-4 Turbo 54% 100% 100% 6.1 GPT-4.1 mini 28% 56% 98% 4.9 Claude Sonnet 3.5 0% NA 100% 12.5 Claude Sonnet 4 0% NA 40% 79.3 Prompt injection (Table 2, AgentDojo 56 samples): Ours achieves 75.0%, 87.5%, 75.0%, 55.4%, 62.5% on Llama-3.1-70B-Instruct, GPT-4o mini, GPT-4o, Claude Haiku 3.5, Claude Sonnet 3.7 respectively, versus 42.1%, 32.1%, 23.2%, 28.6%, 26.8% before optimization; with logprob is 61.4%, 85.7%, 76.8% on the first three models, and the Claude models do not provide logprob access (NA).
- The authors state that the method exceeds the before-optimization baseline on all models, and is comparable to logprob optimization on models where logprobs are available; for GPT-4.1 mini jailbreaking, the logprob method (56%) is actually lower than the method in this paper (98%), which the authors suggest may be because directly optimizing affirmative responses triggers defenses, whereas comparison queries appear harmless.
- Claude Sonnet 4 jailbreaking ASR is only 40%, the lowest in the main jailbreaking table.
Adversarial examples (Table 1, Figure 5)
- Untargeted attacks: text-only queries (Ours) achieve ASR of 5.3%–50.8% across the 10 models, lowest Qwen2.5-VL-3B-Instruct 5.3%, highest GPT-4o mini 50.8%; Transfer-only is 31.6%–92.9%; Transfer+ours is 59.6%–94.7%; Ensemble is 59.6%–96.5% (Table 1).
- Hybrid initialization: the authors state that Transfer+ours improves over both individual methods, e.g., Llama-3.2-11B-Vision rises from 26.3% (Ours) and 57.9% (Transfer-only) to 71.9%; but Qwen2.5-VL-3B-Instruct shows no improvement, which the authors attribute to the model always choosing the first candidate (even when the order is swapped).
- Targeted attacks: unsuccessful on Qwen and Llama (the authors state these models cannot complete complex comparisons); in Figure 5, GPT-4o mini rises from 66.6% (Transfer-only) to 70.1% (Transfer+ours), GPT-5 mini from 70.1% to 79.0%, Claude Haiku 3.5 from 38.5% to 42.1%.
Calibration validation and logprob baseline
- Signal validation (Figure 3): on Qwen2.5-VL-72B-Instruct, absolute confidence queries output only values such as 0, 5, 95; binary comparison answers are on average calibrated with the logit gap, with more false negatives than false positives.
- logprob baseline (Table 4, Figure 6): with logprobs, untargeted ASR rises substantially (e.g., Llama-3.2-90B-Vision from 10.5% to 80.7% for Transfer+logprobs); targeted attacks on the Qwen and Llama families succeed only with logprobs (Figure 6: Qwen-3B 19.3%→71.9%, Llama-90B 86.0%→100.0%, both Transfer-only→Transfer w/ logprob).
Other ablations and analyses
- Jailbreak templates (Table 5): random suffix vs strong template—Llama-3.1-70B-Instruct both 100% (16.6 vs 15.2 queries); Claude Sonnet 3.5 78% (27.9 queries) vs 100% (12.5 queries).
- Failure modes: GPT-4o mini persistently refuses jailbreak comparison queries and cannot provide a signal; Qwen2.5-VL-3B-Instruct always selects the first candidate.
- Appendix D gives examples of optimization trajectories for the three attacks (including an OpenAI Playground screenshot, Figure 10).
What are some points that can be further explored?
The authors list three failure modes, tuning and optimization improvements not done, and defenses discussed but not empirically tested.
Limitations and future directions noted by the authors
- Failure modes (Discussion): weaker models cannot meaningfully distinguish candidates (e.g., Qwen2.5-VL-3B-Instruct always selects the first); comparisons have false positives and false negatives, affecting optimization efficiency; strongly aligned models may completely refuse the comparison task (e.g., GPT-4o mini persistently refuses during jailbreaking).
- No tuning (Discussion): the authors state that the main goal is to verify that LLM-expressed confidence can serve as a signal; they did not do extensive tuning or model-specific adaptation, nor explore other perturbation methods.
- Future directions (Discussion): combine multiple prompts to improve stability; use multiple queries per update with majority voting as an ensemble; more complex input optimization algorithms; develop general, transferable adversarial inputs.
- Defenses (Discussion, discussed only, not evaluated): extend safety alignment so models refuse to provide feedback that can guide iterative attacks; restrict confidence expression for safety-sensitive queries on the API side; deploy systems that detect consecutive similar queries (citing Chen et al., 2020); the authors state that a balance is needed to avoid harming legitimate uses.
Scope of experiments
- 17 models tested in total: 10 vision-LLMs for adversarial examples, 5 LLMs for prompt injection, 6 LLMs for jailbreaking (four families: Llama, Qwen, GPT, Claude).
- Dataset sizes: 57 ImageNet images, 56 samples adapted from AgentDojo, 50 AdvBench items.
- Jailbreaking judging uses two judges, Llama-3.1-8B and GPT-4o mini; the paper does not report agreement between judges and human judgment.
- The paper does not report the number of repetitions per condition; prompt injection and adversarial examples do not report average query counts (the abstract states an average of 5–450 queries across the three scenarios).
- Defenses are discussed only in the Discussion and were not empirically evaluated.
Summarize the main content of the paper
Black-box adversarial optimization using the target model's own binary comparisons, effective in three scenarios, with stronger models more easily attacked.
A black-box attack paper: in a setting where the target LLM returns only text, with no logits and no surrogate model, it obtains an optimization signal for hill-climbing by having the model compare which of two candidate inputs is closer to the attack goal.
- Problem: under text-only APIs, existing attacks can only rely on surrogate signals such as transfer or external reward models; directly asking for absolute confidence collapses to a few values and may be refused.
- Method: at each step, randomly perturb the current input (for images, the square ℓ∞ perturbation of Square Attack; for text, randomly replace suffix segments), submit a binary comparison query to the model, and accept if preferred; applied to adversarial examples, prompt injection, and jailbreaks.
- Jailbreak (Table 3): AdvBench 50 items, budget 200 queries, ASR 100% on Llama-3.1-8B/70B, GPT-4 Turbo, and Claude Sonnet 3.5, 98% on GPT-4.1 mini, 40% on Claude Sonnet 4; average queries 4.9–79.3.
- Prompt injection (Table 2): AgentDojo 56 samples, ASR rises from 23.2%–42.1% before optimization to 55.4%–87.5%, comparable to logprob methods, and the only feasible optimization route on Claude models.
- Adversarial examples (Table 1): ImageNet 57 images, text-only query ASR 5.3%–50.8%, and with transfer initialization the Ensemble reaches 59.6%–96.5%.
- Authors' conclusions: The authors state that larger, stronger models have better-calibrated comparison confidence and are therefore more vulnerable, constituting a safety paradox of "the more capable, the more vulnerable"; the authors believe comparison queries appear harmless and are unlikely to trigger defenses. The authors disclosed to OpenAI and Anthropic before submission.