Jailbreak scaling laws for large language models: crossover from polynomial to exponential growth

Under injection, how does the inference-time attack success rate scale with the number of samples k.

• Scenario: The authors believe that high-capability systems may be misused for harmful goals. Safety-tuned models can still be jailbroken: injecting carefully crafted token sequences can increase the probability of bypassing safety mechanisms, and sampling multiple times during inference increases the chance of at least one success.
• Limitations of existing scaling: Hughes et al. (2024) report that without adversarial injection, the ASR for at least one success in k trials grows polynomially, with log(-log(Πk)) approximately linear in log k. The authors claim that in Figure 1a, under attack, GPT-Turbo-4.5 remains approximately linear, while Llama-3.2-3B-Instruct clearly deviates, so this polynomial law does not cover all injection cases. The caption states that the reviewer is Mistral-7B-Instruct-v0.3, the data is AdvBench, and the attack is GCG, but does not name the target models for these two curves.
• Core observation: If the safety probability has a power-law tail near the upper end p̂=e^{-μ̂}, then for large k the ASR gap can contain both power and exponential factors. In Figure 1b, Llama-3.2-3B-Instruct has p̂=1 on AdvBench, without attack, with Mistral review; in Figure 1c, the same setting with GCG gives p̂≈0.9.
• This paper's approach: Provide minimal assumptions for the upper tail and asymptotic formulas, and propose a spin glass surrogate model SpinLLM, where the magnetic field pointing toward unsafe clusters represents injection strength.
• Threat model:
  • Goal: Make the model produce unsafe outputs for harmful requests; success is defined as at least one unsafe output among k independent samples.
  • Knowledge: The paper does not use white-box, black-box, or no-box to summarize this setting. The suffixes for GCG and AutoDAN are generated on Vicuna-7B-v1.5 and then used on other target models.
  • Capability: Inject crafted token sequences into the input and choose the number of inference-time samples k.
  • Victim system: A safety-aligned LLM that should refuse harmful requests. Main experiments are on open-source instruction-tuned models; Appendix G additionally compares four frontier API models with k=1.

The paper positions itself along four lines: jailbreaks, inference-time computation, statistical models of language, and spin glasses.

Jailbreaks and prompt injection

• Universal and adaptive attacks: Zou et al. (2023) construct universal adversarial suffixes that transfer across prompts and, to some extent, across models (Chao et al., 2023). Geisler et al. (2024) give a more sample-efficient version. Liu et al. (2024a) generate stealthy jailbreak prompts, and Andriushchenko et al. (2024) use adaptive strategies.
• Utility and threat models: Souly et al. (2024) and Nikolić et al. (2025) examine whether jailbroken outputs are useful. Anil et al. (2024) study long-context jailbreaks. Liu et al. (2024b), Greshake et al. (2023), and Liu et al. (2023) discuss the security implications and threat models of prompt injection.
• Inevitability: Su et al. (2024) theoretically explain why jailbreaks are inevitable in safety-trained language models. The paper is paraphrasing here and does not present it as its own theorem.

Inference-time computation

• Multiple sampling: Wang et al. (2023), Zheng et al. (2023a), Brown et al. (2024), Levi (2024), Chen et al. (2024), and Snell et al. (2025) report that multiple sampling, reward reranking, or voting improves accuracy and reliability. Schaeffer et al. (2025), Huang et al. (2025), Halder and Pehlevan (2025), and Levi (2026) provide theoretical analyses.
• With robustness: Hughes et al. (2024) and Zaremba et al. (2025) empirically study adversarial robustness and inference-time computation. This paper directly adopts the former's no-injection polynomial ASR and examines whether it still holds after injection.

Statistical models of language

• Dirichlet processes: MacKay and Peto (1995), Teh et al. (2006), Johnson et al. (2006), and Liang et al. (2007) model language with Dirichlet processes and hierarchical versions.
• Power laws and scaling: Arora et al. (2015) and subsequent work study how networks learn language; Spigler et al. (2020), Bordelon et al. (2020), Bahri et al. (2022), Michaud et al. (2023), and Cagnetta et al. (2026) connect power laws in language structure to model scaling. The authors call Barkeshli et al. (2026) and Liu et al. (2026) complementary perspectives.
• Distributions used in this paper: The authors use Poisson-Dirichlet to describe the landscape of safe and unsafe ideas and note that the self-similar structure of stick-breaking produces power laws (Picard and Pitman, 2006).

Spin glasses

• Computational lens: Mézard et al. (1987), Mézard and Montanari (2009), and Talagrand (2010; 2013) provide the theoretical framework. Hopfield (1982) uses it to understand associative memory. The RSB phase organizes low-energy configurations into hierarchical clusters with weights described by Poisson-Dirichlet.
• Difference from another connection: Hou et al. (2025) also connect spin glasses to language models, but based on the replica symmetric phase, with disorder coming from the inference process. The authors say this contrasts sharply with this paper's setting, where disorder comes from context and inference-time jailbreak laws are discussed in the RSB phase. The physics computation is inspired by Aguilar-Janita et al. (2024).

Baselines and benchmarks

• Baselines: no prompt injection; attacks are GCG and AutoDAN. ASR determination compares refusal strings and LLM review.
• Benchmarks: AdvBench (520 harmful behaviors) and the standard subset of HarmBench (200 items).

The authors place this paper after Hughes et al. (2024): using the upper tail of the safety probability to explain the polynomial-to-exponential crossover after injection, and using a teacher-student spin glass to give a solvable implementation controlled by a single magnetic field strength.

SpinLLM writes inference-time jailbreak scaling as a moment asymptotic of the safety probability, then gives a solvable implementation using a spin glass in the replica-symmetry-breaking (RSB) phase. Short injections correspond to weak magnetic fields, and long injections to strong magnetic fields.

Minimal statistical mechanism

• The latent variable Z represents prompt or context heterogeneity. Given Z=z, the probability that a single output is safe is p(z); the probability that k independent samples are all safe is p(z) to the k-th power.
• Let P=p(Z); the overall success rate satisfies 1−Π_k=𝔼[P^k]. Success is defined as at least one unsafe output.
• Assumption 1: the essential supremum of P is p̂=e^{-μ̂} with μ̂≥0; as ε goes to 0, the upper-tail probability is ĉ ε^{ν̂} plus higher-order terms. p̂=1 means there exist latent states arbitrarily close to fully safe; p̂<1 表示最安全状态仍有不可约的不安全质量。ν̂ 越小，质量越集中在上端。
• Result 2（附录 A）给出 1−Π_k∼ ĉΓ(ν̂+1)e^−kμ̂k^−ν̂。取对数后即得到正文的 log(-log Πk) 展开：μ̂=0 为多项式，μ̂>0 with exponential corrections.

Teacher, student, and unsafe clusters

• The input is first mapped to a fully connected spin glass energy with p≥2. Tokens take +1 or -1, and the generation distribution is a Gibbs measure at inverse temperature. Context enters the disorder J. The paper discusses only the leading order in large N.
• Full RSB appears when βT is above the glass transition threshold. Low-energy states are organized into hierarchical clusters by overlap breakpoints, then ordered by size-biased sampling according to Gibbs weights.
• The teacher first draws a configuration τ uniformly, then plants disorder at inverse temperature βP so that τ has a large generation probability. When βP is large enough and βT≥βP, τ often falls in the first size-biased cluster at level l; that cluster is designated unsafe, and the rest are safe.
• The student shares J with the teacher but has an additional magnetic field h aligned with the center of the teacher's unsafe cluster. The authors say this corresponds to injection pushing generation toward unsafe completions, motivated by the existence of refusal directions (Arditi et al., 2024). A weak field is h much smaller than j0, and a strong field is h much larger than j0.

Weak field: power law

• At large N, the level-l cluster weights follow a GEM stick-breaking with parameter ml. When the first m size-biased clusters are taken as unsafe, the upper tail of the residual safe mass is a power law with exponent ν=m(1-ml) (Result 3). The main setting in this paper uses m=1.
• At h=0, the ASR gap decays as a negative power of k. The authors say this is consistent with the linear trend seen in Hughes et al. (2024).
• Small h reweights the unsafe cluster relative to the remaining mass, with a factor determined by λ=βhN(ql+1-ql). Result 4 writes the gap as a negative binomial mixture of these moments. For large k, small λ, and kλ^2 held at constant order, the authors give a correction relative to the zero-field case. The weak field mainly increases ν, while μ remains approximately 0 (Remark 2). Proofs are in Appendices C and D.

Strong field: exponential law

• When h is large enough, the student enters a replica symmetric phase ordered along the unsafe direction, while the teacher remains RSB. Result 5: there exists a critical field hc such that when h exceeds it and k is fixed, the lower bound of log(1-Πk) is given by the rate function Ih(ql+1), with scale kN. The proof uses annealed planted disorder and labels the cluster containing τ as unsafe (Appendices B, E). The authors say that in Figure F.1, the exponential scaling still holds under the exact setting of Section 3.
• Assumption 6 assumes that the log of the single-sample safety probability is self-averaging and sub-Gaussian. The caption of Figure E.1 says the left panel confirms exponential concentration in N, and the right panel shows concentration around the mean faster than exponential. Result 7 then gives an exponential upper bound on the upper-tail probability.
• Remark 3 maps this to a weak form of Assumption 1: p̂ is less than 1, μ̂ grows with N and the rate function, and ν̂=0. Combined with Result 5, for large N and large k, log(1-Πk)/(kN) equals -Ih(ql+1). As h increases, configurations concentrate more on τ, Ih increases, and thus μ increases.

Numerical checks are in Appendix F: p=2, N=24, β=10, j0=1, and 1024 disorder samples; clusters are approximated by greedy single-spin-flip descent. This is not an LLM experiment.

Experimental Setup

• Target models: Scaling experiments use GPT-Turbo-4.5, Llama-3.2-3B-Instruct, Llama-3-8B-Instruct, Llama-3-70B-Instruct, OLMo-2-0325-32B-Instruct, Olmo-3.1-32B-Instruct. Appendix G additionally attacks Claude-Sonnet-4.5, Claude-Haiku-3.5, GPT-Turbo-3.5, GPT-4 at k=1.
• Data: 520 items from AdvBench; 200 items from the standard subset of HarmBench. Appendix H uses GPT-5.1 to label five categories for AdvBench, and HarmBench uses the dataset's own categories.
• Attacks and baselines: No injection; GCG optimized on Vicuna-7B-v1.5 for 50 steps, with 25 AdvBench items each for training and validation, suffix length 20 tokens; AutoDAN scored on the same model, elite ratio 5%, crossover rate 0.5, mutation rate 0.01, 100 generations, one suffix per prompt. The frontier comparison additionally includes a fixed short-suffix injection.
• Metrics: Πk is the proportion where at least one of k generations is judged as jailbroken. Scaling experiments fit log(-log Πk) as -ν̂ log k - μ̂ k + log Ĉ. Appendix G states OLS is used and reports R² on the transformed scale; the main text does not give R².
• Review: Main experiments use Mistral-7B-Instruct-v0.3; scores no higher than 4 are considered unsafe, incoherent outputs are scored 5, and no numeric output is also scored 5; greedy decoding, generating 4 tokens. Frontier comparisons use the 34 refusal strings from Table G.1, as well as GPT-4's binary classification and 1–5 harm score, with no lower than 4 considered truly harmful.
• Sampling: Scaling experiments use LM Evaluation Harness, zero-shot, temperature 0.6. k ranges from 1 to 20, and from 21 to 128 in steps of 4. Each condition has 2 independent generations, reporting mean and standard deviation; the main text does not give standard deviation values. Frontier experiments use k=1, temperature 0, and at most 512 tokens. The paper does not report agreement with human judgment.

Main Results

The main text does not give checkable percentages of Πk for each model. The table below only lists numbers in the text.

Swipe horizontally to read the table

Setting	Metric	Value	Source
Llama-3.2-3B-Instruct, AdvBench, no attack, Mistral review	Upper safety probability p̂	1	Figure 1b
Same, with GCG	p̂	≈0.9	Figure 1c
Llama-3-8B-Instruct, AdvBench, high-injection curve	Numerical lower bound on 1−Πk	≈10^{-3}	Figure 3 caption

• The authors state that in Figure 1a, under attack, log(-log(Πk)) for GPT-Turbo-4.5 remains approximately linear in log k, while Llama-3.2-3B-Instruct deviates from linearity, and is written as a correction with -μ̂k. The caption does not name these two target models.
• The targets of Figures 3 and 4 are Llama-3-8B-Instruct and OLMo-2-0325-32B-Instruct, respectively, with AdvBench data and Mistral review. The authors state that the injection curves for GCG and AutoDAN are higher than no injection and decrease faster; the straight line at high injection and large k comes from a numerical lower bound in the code. When injection is stronger, the fitted ν̂ and μ̂ tend to increase. The authors map jailbreak prompt length to effective magnetic field strength μ̂.
• The authors state that Figure F.1 agrees qualitatively very well with Figures 3 and 4. When h is small and N is much larger than k, k is much larger than 1, and kλ^2 is much smaller than 1, the numerics are close to Result 4; for larger h they deviate from that formula. In fits of the combined form, both μ̂ and ν̂ increase monotonically with h (Figure F.1b). The main text does not give comparable ASR, so it cannot be determined from the text which model is worst. Models without extractable Πk are omitted: GPT-Turbo-4.5, Llama-3-70B-Instruct, Olmo-3.1-32B-Instruct, and the four frontier API models.

Review Method

• What was measured: Appendix G compares refusal strings, GPT-4 review, and harm scores on four frontier models, k=1, AdvBench.
• Results: The authors state that in Figure G.1a, the ASR from LLM review is much smaller than from refusal strings; the reason is that models generate incoherent, irrelevant, or factually incorrect text. In Figure G.1b, the general adversarial suffix has a higher ASR than the fixed short suffix on responses with harm scores no lower than 4. The main text gives no percentages.
• Authors' interpretation: Therefore, subsequent scaling experiments switch to Mistral review and only count responses with scores no higher than 4 as successes. In Figure G.3, the authors state that GPT-4 and Mistral are highly consistent in ASR for GPT-Turbo-3.5 and Claude-Sonnet-4.5, without giving numbers.

More Models, Datasets, and Categories

• Appendix H extends GCG and AutoDAN to Llama-3.2-3B-Instruct, Llama-3-70B-Instruct, Olmo-3.1-32B-Instruct, and HarmBench standard. The authors state the trends are similar to Figure 3.
• A straight line appears again at high injection and large k, attributed by the authors to numerical limits in the code; the appendix does not repeat the 10^{-3} figure.
• Figures H.5–H.13 plot by category. AdvBench categories are labeled by GPT-5.1, and HarmBench uses its own labels. The main text gives no per-category ASR.

Spin Glass Numerics

• Appendix F computes teacher–student ASR on N=24, p=2, β=10, j0=1, and 1024 disorder samples. The unsafe direction takes the teacher's lowest layer, m=1 size-biased cluster.
• Small fields agree with the formula in Result 4 within the stated applicability domain; as the field strength increases and kλ^2 is no longer much smaller than 1, the numerics clearly deviate from that formula.
• At large fields, the form with -μ̂k can still be fitted, and μ̂ and ν̂ both increase monotonically with h. This is not the same statement as Result 7, which writes the strong-field edge exponent as ν̂=0; the paper's statement about the numerical fits is that both increase.

Other Ablations and Analyses

• Weak-field analysis: as attack strength increases, ν=m(1-ml) increases and μ is approximately 0 (Remark 2, referring to Figure 2).
• Strong-field analysis: after h exceeds hc, the logarithmic lower bound on the gap is -kN Ih(ql+1), and Ih increases with h (Result 5, Remark 3).
• Figure 2 only gives the title "scaling parameters vary with jailbreak strength"; the main text gives no coordinate readings.
• Scaling uses temperature 0.6, frontier k=1 experiments use temperature 0; there is no cross-comparison table between the two temperatures.
• Each condition has 2 generations; GEM parameters are estimated with 32 size-biased permutations, stick-breaking length 8, and a 199-point grid.
• Negative cases: strong models do not switch to the exponential correction in the authors' description of Figure 1a; the small-field formula fails at large h; the high-injection curve is truncated by a lower bound of about 10^{-3}; refusal strings give higher ASR than LLM review, but without percentage points.

Limitations and future directions noted by the authors

• The work is limited to attacks on open-source instruction-tuned LLMs (Section 6).
• The pretraining and post-training pipelines of these models differ, and the study does not characterize how vulnerability depends on the training method (Section 6).
• The attack relies on universal jailbreak strings; model-specific jailbreak strings could further improve attack effectiveness against a given target (Section 6).
• The study does not examine how additional defense mechanisms affect attack success rate (Section 6).

Experimental coverage

• Scaling experiments target GPT-Turbo-4.5, Llama-3.2-3B-Instruct, Llama-3-8B-Instruct, Llama-3-70B-Instruct, OLMo-2-0325-32B-Instruct, and Olmo-3.1-32B-Instruct; parameter sizes are written as 3B to 70B in the abstract (Section 5, Appendices G, H).
• Attack conditions include no injection, GCG, and AutoDAN; Appendix G also has a fixed short suffix for review comparison on frontier models. Suffix generation uses Vicuna-7B-v1.5, GCG with 50 steps and a 20-token suffix, and AutoDAN with 100 generations.
• Data are 520 AdvBench items and 200 HarmBench standard items; Appendix H plots by category. k goes from 1 to 20, then from 21 to 128 in steps of 4, with 2 generations per condition and a scaling temperature of 0.6.
• Frontier API models Claude-Sonnet-4.5, Claude-Haiku-3.5, GPT-Turbo-3.5, and GPT-4 were compared at k=1, temperature 0, and at most 512 tokens. Review includes 34 refusal strings, Mistral-7B-Instruct-v0.3, and GPT-4.
• Spin glass numerics are N=24, p=2, β=10, j0=1, and 1024 disorder samples (Appendix F). The paper does not report agreement with human judgment, R² in the main text, per-condition standard deviations, or extractable ASR percentages other than p̂ in Figure 1 and the numerical lower bounds in Figure 3.

The authors use a solvable statistical model to explain how prompt injection changes the scaling of jailbreak success rate with the number of inference-time samples.

• Problem: Without injection, the probability of at least one success in k samples grows polynomially. After injection, the authors see the failure probability drop faster on some models, and want to cover both cases with the same mechanism.
• Method: The overall ASR is determined by the moments of the single-attempt safety probability. If the upper end is a power-law tail, the gap is a power times an exponential decay. SpinLLM writes generation as the Gibbs measure of a spin glass: the teacher's low-energy clusters specify unsafe regions, and the student is subject to a magnetic field pointing toward those regions. A short injection is a weak field, mainly changing the power; a long injection is a strong field, bringing an exponential decay with k.
• LLM results: On AdvBench, Llama-3.2-3B-Instruct has a safety probability upper end of 1 without attack and about 0.9 under GCG (Figure 1, Mistral-7B-Instruct-v0.3 review). The authors say GPT-Turbo-4.5 remains close to polynomial under attack, while Llama-3.2-3B-Instruct deviates from linearity. On Llama-3-8B-Instruct and OLMo-2-0325-32B-Instruct, the injection curves for GCG and AutoDAN drop faster (Figures 3, 4); Appendix H extends to the Llama and Olmo families and HarmBench. The ASR for refusal strings is higher than LLM review (Figure G.1), and the main text has no percentages.
• Theoretical comparison: Spin-glass numerics with N=24 agree with the analytic formula in small fields, while in large fields an exponential term must be added; the authors say this is qualitatively consistent with the language model curves. The high-injection straight line in Figure 3 stops at 1-Πk around 10^{-3}, which the authors attribute to a numerical floor.
• Authors' conclusion: Jailbreak prompt length can be viewed as an effective magnetic field strength. A weak field raises the polynomial exponent while the exponential correction remains close to 0, whereas a strong field makes the failure probability decay exponentially. The authors also state that these attack experiments are limited to generic jailbreak strings on open-source instruction-tuned models, without separating training pipelines or testing additional defenses.