A systematic scaling analysis of large language model jailbreak attacks

Core question: how jailbreak attack success changes with the compute invested by the attacker, and whether this can be characterized and compared across methods like language model scaling laws.

• Setting and motivation: The authors note that language modeling has mature scaling laws for the relationship between performance and compute, data, and parameters, but it is unclear whether jailbreak ASR or judge scores follow a similar regularity; preliminary experiments show that attack success first rises and then plateaus with investment.
• Shortcomings of existing analyses: The authors argue that existing "attack scaling" analyses are either hard to map to realistic jailbreak threat models (e.g., assuming direct control of internal activations, evaluating on synthetic goals) or cover only a small slice of attack design (e.g., a specific black-box adversarial setting or ensemble-style protocol).
• Comparability obstacle: Each iterative attack defines a "step" differently (a gradient update, an LLM rewrite round, a population generation, a random sample each count as one step), so raw iteration counts are not comparable; the authors therefore adopt a unified FLOPs compute axis.
• Approach of the paper: Place four representative attack paradigms (optimization-based, self-refinement rewriting, sampling-and-selection, genetic optimization) on a shared FLOPs axis, fit saturating exponential curves, and compare efficiency, stealthiness, and differences across goal categories.
• Threat model (prompt-only, Section 3.1):
  • Goal: Construct an input prompt such that the victim model produces output that both violates the safety policy and substantively responds to the malicious goal g.
  • Knowledge: Divided into black-box (query access only, observing sampled outputs) and white-box (access to parameters or equivalent gradient/logit signals); in the evaluation only GCG is white-box, while PAIR, BoN, and AutoDAN are black-box.
  • Capabilities: Control of the initial adversarial prompt, the attack procedure and hyperparameters, and an optional auxiliary attack model; no modification of the victim model's weights or decoding hyperparameters.
  • Victim system: A safety-aligned LLM with a fixed system prompt.

Jailbreak and Automated Red-Teaming Methods

• Optimization-based jailbreaks: GCG (Zou et al., 2023) performs adversarial suffix search; Basani and Zhang (2025) perform black-box suffix generation.
• Iterative black-box refinement: PAIR (Chao et al., 2024b) is driven by an attacker LLM rewriting; TAP (Mehrotra et al., 2024) extends it to tree search; Yang et al. (2025b) state that most multi-turn jailbreaks can be reduced to repeated sampling.
• Sampling and evolutionary search: best-of-n (Hughes et al., 2024) relies on large-scale sampling to select the best; AutoDAN (Liu et al., 2024) and subsequent work use genetic algorithms, reinforcement learning, or variational inference to improve efficiency or diversity.

Evaluation and Judging Methods

• Benchmarks: HarmBench (Mazeika et al., 2024) and JailbreakBench (Chao et al., 2024a) provide standardized evaluation; this paper's evaluation set mixes AdvBench-Harmful Behavior, HarmBench, and ClearHarm, each about one third, for 200 objectives in total.
• Success judging: early work used target string matching (Zou et al., 2023), while recent benchmarks have shifted to LLM semantic judging to reduce false positives; this paper follows this line, using GPT-5 judged scores as the primary metric.

Scaling Laws of Adversarial Behavior

• Classical scaling laws: Hestness et al. (2017), Kaplan et al. (2020), and Hoffmann et al. (2022) characterize the relationship between performance and compute/data/parameters; this paper borrows their "performance–resource" relationship to compare attack families.
• Attack-side scaling: Liu et al. (2025) analyze how black-box adversarial attacks change with model ensemble size; Panfilov et al. (2026) study how red-team success changes with attacker model capability; Fort (2023) finds that controllable output length grows linearly with the number of tokens controlled by the attacker. The paper points out that Fort (2023) assumes direct access to activations and evaluates non-natural token sequences, which does not constitute a realistic jailbreak threat model.

Positioning of This Paper: the authors state that, unlike the above work, this paper targets realistic jailbreak prompts with natural language input, and systematically compares four attack paradigms, multiple model families and sizes, and multiple harm categories on a unified FLOPs axis.

Overall idea: treat each jailbreak attack as a compute-bounded optimization process, use total FLOPs as a unified resource axis, measure the trajectory of judge scores as compute increases, and fit and compare them with a saturating exponential model. The framework has no separate name; the core is mapping an attack–model–dataset triple to a scaling curve.

Problem formulation

• Scaling curve: For each (attack A, model fθ, dataset D) and budget B, define S_A,θ,D(B) = ASR(A, f_θ, D; B), i.e., the average judge score under that budget; a relevance score is also tracked as a supplementary measure of content similarity.
• Efficiency measure: Define the compute required to reach a threshold τ as B(τ) = min{B : S(B) ≥ τ}, used for cross-attack comparison.

Data and categories

• Evaluation set: A mixture of three public benchmarks, AdvBench-Harmful Behavior, HarmBench, and ClearHarm, each about one third, for a total of n = 200 goals; 200 is fixed to balance coverage and total compute cost.
• Goal categories: Each goal is labeled as one of four categories: harmful_instruction, malicious_creation, misinformation, offensive, using a keyword-based rule classifier with priority matching (the keyword list is in Appendix A.5).

Attacks and models

• Four attacks: GCG (discrete gradient optimization, with the suffix wrapped in a fixed template borrowed from Andriushchenko et al. (2025)), PAIR (iterative rewriting by an attacker LLM), BoN (sample and select the best), AutoDAN (genetic algorithm).
• Models: The main comparison uses Llama-3.1-8B-Instruct; within-family size scaling uses Qwen3 at 1.7B/4B/8B (non-thinking); cross-family comparison adds Gemma-3-4B and Llama-3.2-3B-Instruct at roughly the 4B tier. Reasoning/thinking variants are excluded; the authors state that their behavior is more unstable and unfavorable for stable compute scaling comparisons.

Metrics

• Success: GPT-5 judges output a red-team score in [1,10], assessing both whether the safety policy is violated and whether the malicious intent is substantively addressed; the main metric ASR is the average score over the evaluation set (the judge prompt is in Appendix A.1). There is also a relevance score in [0,10], which measures only semantic similarity and does not judge violation (Appendix A.2).
• Compute: Counts the victim model's forward and backward passes and the auxiliary attack model's forward calls, converted to FLOPs based on input/output token lengths.
• Stealthiness: Uses GPT-2 perplexity, normalized to a stealth score in [0,1] by the minimum/maximum perplexity in the evaluation logs; lower perplexity is more "stealthy".

Curve fitting and mechanism analysis

• Fit form: ASR(B) = a + b (1 − e^−cB), where a is the starting point, a+b is the asymptotic upper limit, and c controls the rate of approach; fitted by nonlinear least squares on FLOPs-aligned points.
• Efficiency statistics: Closed-form derivations B50 = ln 2/c and B95 = −ln(0.05)/c, denoting the compute required to reach 50%/95% of the available improvement space.
• Mechanism analysis: Map PAIR's rewriting to an optimization perspective, and under GCG's objective loss L(x) = −log p_θ(y^⋆ ∣ x) perform a same-state single-step comparison: starting from the same state xt on the PAIR trajectory, compare the loss change, step size, and direction consistency of PAIR's actual step versus GCG's suggested step; a goal-matched PAIR variant is also set up to rule out the explanation of goal mismatch.

Experimental Setup

• Models tested: Llama-3.1-8B-Instruct (main comparison); Qwen3-1.7B/4B/8B (within-family sizes); Gemma-3-4B, Llama-3.2-3B-Instruct (cross-family at roughly the 4B tier).
• Dataset: a mixed set of 200 objectives, with AdvBench-Harmful Behavior, HarmBench, and ClearHarm each accounting for about one third; divided by rule into four objective categories.
• Attack methods: GCG, PAIR, BoN, AutoDAN; GCG is white-box, the rest are black-box.
• Metrics and judging: GPT-5-judged red-team score ([1,10], primary metric) and relevance score ([0,10]); stealth score normalized by GPT-2 perplexity; compute measured in cumulative TFLOPs.
• Budget grid: each (attack, model, split) is evaluated over a series of compute budgets from low to saturation, with the main figure window at 0–20K TFLOPs; the paper does not report the number of repetitions per budget point.

Main Results

Fitted parameters of Eq. (7) on Llama-3.1-8B-Instruct (per Table 1; a is the starting point, a+b the upper limit, B50/B95 in TFLOPs):

Swipe horizontally to read the table

Attack	red-team a	red-team a+b	B50	B95	relevance a+b
GCG	3.19	5.16	3221	13923	1.46
BoN	3.79	8.83	1322	5715	5.38
PAIR	9.11	9.80	1391	6012	3.86
AutoDAN	7.17	8.92	2848	12307	3.00

• The authors state that all four attacks show a fast-then-flat saturation shape, fitted reasonably well by the saturating exponential model (R² of 0.804–0.997).
• The authors state that PAIR has the highest starting point, the highest upper limit, and saturates quickly, while GCG has a low upper limit and grows slowly; under the relevance score the ordering changes, with BoN's upper limit overtaking PAIR, which the authors attribute to BoN making only surface-level perturbations and preserving the original semantics.
• The authors point out that relevance upper limits are generally lower than red-team upper limits, indicating that responses judged as violations often do not substantively address the attack objective.

Mechanistic Analysis: Why PAIR Is More Efficient Than GCG (Section 5.3)

• Objective-matching ablation: changing PAIR's attacker LLM to induce the exact target string y*, its asymptotic upper limit is still 8.60, higher than GCG's 5.16; the authors therefore argue that the efficiency gap cannot be explained by objective mismatch alone.
• Same-state single-step comparison: starting from the same prompt state, the step length of a standard GCG single step is much smaller than PAIR's, and the loss decrease is very small; after changing GCG to modify 30 tokens per step so that step lengths are comparable, the directional improvement of GCG's suggested steps is almost entirely positive (i.e., loss increases), and the cosine similarity of the two update directions is close to zero or negative (Table 2).
• The authors' interpretation: PAIR's advantage mainly comes from more reliably finding effective update directions in prompt space, and remaining directionally effective at larger step lengths.

Stealth and Objective Categories

• ASR–stealth operating points (Figure 3): the authors state that PAIR lies in the high-success, high-stealth region; AutoDAN is highly stealthy but has lower success; BoN has high success but lower stealth; GCG's perplexity is pulled down because the suffix is wrapped in a fixed template, and the raw suffix (GCG(orig)) is far less stealthy.
• Objective categories (Figure 4, GCG broken down by category, window up to 200K TFLOPs): categories differ not only in absolute scores but also in marginal returns, with saturation and occasionally non-monotonic segments; the authors state that misinformation is generally the easiest to induce and has a higher starting point, and suggest this may be related to safety training hardening explicit harmful instructions more.

Other Ablations and Analyses

• Within-family sizes (Table 3a, PAIR on Qwen3): the upper limits of the three sizes cluster at 9.80–9.95; 4B saturates fastest (B50 = 930), 1.7B slowest (B50 = 1093).
• Cross-family (Table 3b, PAIR, roughly the 4B tier): Gemma-3-4B and Qwen3-4B have upper limits of 9.93 and 9.80, Llama-3.2-3B only 7.26, with a starting point of 1.40, B50 = 4303, and B95 = 18595 beyond the evaluation window.
• Authors' conclusion: within-family size mainly affects the starting point and the rate of approach, while cross-family differences can shift both the starting point and the upper limit.

Limitations noted by the authors and future directions

• FLOPs normalization is only an approximation (Section 6): It relies on token counts and model-dependent constants, does not capture deployment constraints such as latency, rate limits, or defender monitoring costs, and the absolute effectiveness of attacks varies by scenario.
• Results depend on the evaluation method (Section 6): Using an LLM judge and a specific rubric may introduce judge bias and sensitivity to prompt wording; the authors consider validation across judge models and on a manually audited subset still important.
• Limited coverage (Section 6): A mixed set of 200 goals, a predominantly English single-turn prompt setting, and fixed model and attack families may not reflect multilingual behavior, longer multi-turn protocols, tool-augmented agents, or other system prompts.
• Future direction one: multi-axis parameterization (Section 6): Extend ASR(B) to ASR(B; model, attack, task, language, protocol) varying with model, attack, task, language, and protocol, using hierarchical models to let (a, b, c) vary with model family/size, goal attributes, and system prompt, supporting partial pooling and uncertainty estimation.
• Future direction two: multi-objective scaling (Section 6): Jointly model attack success and practicality metrics such as detectability and utility loss, so that increasing compute characterizes a predictable trade-off rather than merely higher ASR.

Experimental coverage

• 6 models are tested: Llama-3.1-8B-Instruct, Qwen3-1.7B/4B/8B, Gemma-3-4B, Llama-3.2-3B-Instruct; the main cross-attack comparison is done only on Llama-3.1-8B-Instruct (Section 4.3).
• Attacks cover four types: GCG, PAIR, BoN, AutoDAN; the evaluation set is 200 goals, with three benchmarks each about one third (Section 4.1, 4.2).
• Success determination uses only the GPT-5 judge (red-team and relevance scores), and stealthiness uses only GPT-2 perplexity; the paper does not report experiments on agreement between judging and human determination.
• The paper does not report the number of repetitions per budget point, the specific model of the auxiliary attack model used by PAIR, or the internal hyperparameter details of each attack.
• The paper explicitly excludes model variants in reasoning/thinking mode, on the grounds that the authors state their behavior is more unstable and protocol-sensitive (Section 4.3).

A systematic analysis of the scaling behavior of jailbreak attacks: the authors treat attacks as compute-constrained optimization processes and answer the question "how does jailbreak success change with attack compute, and can different paradigms be compared?"

• Problem: Different attacks define a "step" differently, making iteration counts incomparable, and existing attack scaling analyses either use unrealistic threat models or have narrow coverage.
• Method: On a shared FLOPs axis, measure GPT-5-judged red-team scores (main metric ASR), relevance scores, and GPT-2 perplexity stealth scores, fit a saturating exponential model, and derive B50/B95 efficiency statistics; the evaluation set is 200 goals (three benchmarks each about one third).
• Main results: On Llama-3.1-8B-Instruct, PAIR's red-team fitted upper limit is 9.80 with a starting point of 9.11, while GCG's upper limit is only 5.16 (Table 1); after goal matching, PAIR's upper limit is still 8.60, and the same-state comparison shows its update direction is more effective (Table 2).
• Other findings: The authors state that PAIR occupies an operating point of high success and high stealthiness; Qwen3's three sizes have upper limits concentrated at 9.80–9.95, whereas across families Llama-3.2-3B's upper limit is only 7.26; misinformation-type goals are generally the easiest to induce.
• Authors' conclusion: An ASR snapshot under a single budget does not characterize jailbreak risk well, and reporting compute-normalized scaling curves can better guide red-teaming and defense prioritization; the authors also acknowledge that FLOPs is only an approximation, results depend on judging, and coverage is limited.