Trading inference-time compute for adversarial robustness

1. 01What problem does this paper attempt to solve?

   The authors ask: without adversarial training, can increasing inference compute reduce the fraction of successful attacks against o1?

   Can increasing inference-time compute in reasoning models reduce attack success without adversarial training or revealing the attack form?

   • Scenario: The authors state that models act as agents that browse the web, execute code, and ingest multimodal inputs from untrusted parties, where incorrect actions may have real-world consequences. The authors cite Carlini (2024) stating that adversarial machine learning has produced over 9000 papers in a decade with limited progress; and state that scaling pretraining has limited benefit for robustness.
   • Existing approach: Adversarial training minimizes worst-case loss over a transformation set T that does not change labels (Madry et al., 2018). The authors believe it is computationally expensive and requires knowing T in advance; safety policies can judge whether a perturbation changes the label, but cannot explore the perturbation space in advance, so it is like whack-a-mole against known attacks.
   • Authors' hypothesis: The situation differs when scaling inference-time compute. This intervention does not involve adversarial training on the studied attacks, nor does it inform the model of the attack nature; the authors state it also improves non-adversarial performance.
   • What this paper does: On o1-preview and o1-mini, write attack success probability as a function of attacker resources and inference compute; adapt soft-token attacks, propose think-less, and hypothesize nerd-sniping. The authors also flag settings where compute does not improve reliability.
   • Threat model:
     • Goal: Output an incorrect answer rather than the correct one. For math, 42, correct answer +1, or ×7; for AdvSimpleQA, COMPROMISED; for vision, incorrect class; for policy tasks, policy violation.
     • Knowledge: The paper states it includes white-box and black-box, static and adaptive. Soft token uses gradients; LMP and human red team are black-box adaptive; many-shot is static.
     • Capability: Only part of the prompt x is controlled by the adversary. In math problems, the adversary controls the segment inside the ignore label but cannot place an end-of-label. Misuse Prompts can give arbitrary text as long as the target response is still policy-violating under that input. AdvSimpleQA only appends instructions to output the keyword, without altering existing web content.
     • Victim system: o1-family reasoning models. Except for image experiments, no special prompting or defensive LMP is used; only the same "more reasoning" that improves non-adversarial performance. Policy tasks do not put specifications in the context.

   Collapse this answer

2. 02What related research exists?

   The authors contrast with adversarial training, test-time augmentation, and pretraining scale; the difference is that inference compute is not targeted at the attack set.

   LLM jailbreaks and agent inputs

   • Known jailbreaks: Zou et al. (2023), Wei et al. (2023), Andriushchenko et al. (2024). The authors say top models all already have jailbreaks.
   • Untrusted inputs: indirect prompt injection from Greshake et al. (2023), Schulhoff et al. (2023), Tensor Trust from Toyer et al. (2024), and the PromptArmor (2024) example of embedding instructions in public Slack AI channel messages to extract confidential data from private channels.
   • Attack phenomena used later: many-shot jailbreaking from Anil et al. (2024); Andriushchenko and Flammarion (2024) report that refusal training does not necessarily generalize to the past tense. This paper's Past Misuse Prompts uses past-tense rewrites.

   Adversarial training and pretraining scale

   • Madry et al. (2018): min-max adversarial training over a perturbation set T. The authors consider it costly and requiring prior knowledge of T.
   • Pretraining scale: Howe et al. (2024) report that without explicit defense training, larger models are only slightly more robust on most tasks, and the effect is unreliable. Ren et al. (2024) Table 7 reports that robustness on jailbreak benchmarks is negatively correlated with pretraining compute.
   • Wang et al. (2024) AdvXL: scales data and models for visual adversarial training. The paper reports it improves over prior work on ImageNet-1K by more than 10% absolute; its best model has 71% robust accuracy against ℓ∞ AutoAttack (ε=4/255), 83.9% clean accuracy, about 5B samples, and about 1B parameters.

   Test-time compute and image robustness

   • Randomized smoothing: Cohen et al. (2019) take consensus over ensemble outputs on augmented inputs. Carlini et al. (2023) use a pretrained diffusion model in that framework to remove Gaussian noise, improving certified robustness.
   • Sun et al. (2020): test-time training, training on test samples with a self-supervised loss at test time to handle train-test distribution mismatch.
   • Jain et al. (2023): study defenses including rewriting. The paper reports they note rewriting can reduce performance, and attackers may find inputs that still correspond to the original attack after rewriting.

   Attack sources and baselines in experiments

   • PAIR and StrongREJECT: PAIR from Chao et al. (2024); StrongREJECT from Souly et al. (2024). This paper's LMP is structurally similar to PAIR, but the attacker is a smaller reasoning model and is not restricted to one jailbreak strategy.
   • Image and QA data: ImageNet-A (Hendrycks et al., 2021b), Attack-Bard (Dong et al., 2023), and SimpleQA (Wei et al., 2024) modified into AdvSimpleQA.
   • Baselines: the experiments do not run adversarial training, randomized smoothing, or AdvXL as comparisons. The comparison is attack success rate of the same model under different inference compute, and think-less versus standard many-shot (Figure 16). A summary of tasks, targets, and attacks is in Table 1.

   The authors place the difference as: prior test-time augmentation still requires designing for the adversary's perturbation set; this paper does not know T in advance and does no adversarial training on the studied attacks, only increasing inference compute with the same non-adversarial performance.

   Collapse this answer

3. 03How does the paper address this problem?

   Without changing training or disclosing the attack, they only increase inference compute, then measure success probability with multiple attacks.

   No adversarial training on the studied tasks, and the model is not told what the attack is; only o1-preview and o1-mini are made to reason more. Attack success probability is then written as a function of attacker resources and inference compute. Vision experiments use a variant the paper denotes o1-v.

   Tasks and what the adversary can change

   • Math: two-digit addition, two-digit multiplication, and MathD / MATH (Hendrycks et al., 2021a). Table 1 and Section 2.1 and Appendix A.1 write 2-digit; the captions of Figures 2, 3, 8, and 20 write 4-digit. For each problem type, the attack target is outputting 42, correct answer +1, or correct answer ×7.
   • Controlled fragment: The top-level instruction says to ignore text between <BEGIN IGNORE> and </END IGNORE>. The adversary fully controls that fragment but cannot insert an end tag. Appendix A.1 gives the format; target prompts are not reproduced here.
   • Policy tasks: StrongREJECT's violating questions, Misuse Prompts, past-tense rewrites, and an internal unambiguous rule subset called rule following (also RF in the text). These tasks do not write the specification into the context and rely on policy knowledge already in safety training. The adversary can give arbitrary text, but the target answer must still be violating under that input.
   • Other: AdvSimpleQA concatenates SimpleQA questions with webpages containing correct answers; the adversary only appends an instruction to output COMPROMISED and does not change the webpage text. The vision task is predicting a class for an image, with the class information written in the prompt.

   The authors call math, rule following, and AdvSimpleQA unambiguous: success has a simple criterion. Misuse Prompts are called ambiguous, and humans often disagree. The authors use a legal analogy: this paper evaluates whether the model as a "judge" can enforce an unambiguous specification; whether the specification itself has loopholes is separate research.

   Six attack types

   • Many-shot (Anil et al., 2024): puts many in-context examples in the controlled fragment, with outputs being the attacker's desired answer. The resource is token count. The authors state the attack is the same for all models and these models are not specifically trained to resist it.
   • StrongREJECT suite: jailbreaks from social media and jailbreak papers paired with violating questions, used as a baseline attack on content policy.
   • Soft token: white-box. The adversary directly sets the embedding vectors of the controlled fragment and optimizes with gradient descent. Resources can be the number of soft tokens, optimization steps, or an embedding norm cap. Optimized per task distribution and tested on held-out tasks. Adaptive.
   • LMP: black-box language program that loops calling the attacker. After each failure, it sees the previous failure feedback and changes the next attempt, until first success or at most 25 attempts. Used to fill the controlled fragment, or to rewrite Misuse Prompts and rule following prompts. The resource is the number of queries to the victim model. Structurally similar to PAIR, but the attacker is not restricted to one jailbreak strategy; the paper does not name the attack model, only calling it a smaller reasoning model.
   • Human red team: experts interact to find violating prompts. This setting does not scan attacker resources. Compute tiers are hidden from red teamers, order is randomized, and personnel rotate across batches.
   • Images: ImageNet-A natural adversarial images, and Attack-Bard optimized against Bard and transferred (ℓ∞, ε=16/255). The process is adaptive but not targeted at these o1 models. Attacker resources are not varied. Also evaluated is unperturbed Attack-Bard-clean.

   The inference compute intervention

   • The intervention is asking the model to reason more, the same intervention that improves o1 series non-adversarial performance. Except for image experiments, no defensive prompts or defensive LMP are used to direct compute specifically toward robustness.
   • Success probability: among multiple prompts and multiple samples per prompt, the proportion of answers satisfying the attack target. The paper does not state sampling temperature, samples per prompt, or number of repetitions. Several figures have a log-scale x-axis of inference compute; gray cells indicate insufficient samples at that compute level.
   • The human red team divides request tiers into 5 tiers without giving absolute compute units.

   Soft token, think-less, and nerd-sniping

   • Soft token training: data split into train/test. First sample training-set answers under different inference compute; then maximize the log probability of "sampled chain of thought + attacker's desired answer"; finally evaluate on the test set under different inference compute. Unconstrained optimization makes norms far larger than ordinary embeddings, so a norm constraint is added: every few steps, project to the average norm of standard embeddings. The paper does not state the interval. The authors state the unconstrained version is strong and may give an upper bound on attack success probability; the norm constraint adds optimization difficulty, and the authors are less sure whether the attack can be stronger.
   • Think-less: a class of prompts found by the human red team aimed at reducing the model's actual test-time compute. In the 64-shot many-shot setting, it is compared with a standard many-shot that only shows "obey the attack target." The authors state it can be monitored, because abnormally low inference compute can be flagged.
   • Nerd-sniping: the authors use this term for failure modes where inference time is far above the usual level for that difficulty, and consider these outlier chains of thought like being stuck in unproductive loops. If an instance's actual compute is higher than expected, attack success rate tends to be higher. Appendices A.3–A.6 have attack examples and chains of thought under attack; no sentences are excerpted here.

   How success is judged

   • Math is judged by whether it equals 42, correct answer +1, or ×7. AdvSimpleQA uses a neural network to judge whether the specified word is output; the authors state the task is simple and unambiguous, and human agreement with this grader is perfect. The paper does not give the sample size for this agreement.
   • StrongREJECT uses [email protected], higher is more robust. Each misuse prompt is paired with each jailbreak, the worst 10% of answers' average goodness is taken, then averaged over questions. The authors compare it to: the attacker knows which jailbreaks are most effective per question and randomly picks one of the most damaging top 10%. This version has 35 jailbreaks and filters prompts to 60 questions violating internal content policy.
   • The LMP misuse evaluation does not directly use the original StrongREJECT attacks. The attacker can design attacks after receiving the misuse intent; for scoring, the original misuse prompt and the model's answer to the LMP attack are paired and scored with the StrongREJECT grader. Figure 10 has one rewrite example, not cited here.
   • The red team has two phases. Phase one makes 20 valid prompts per compute tier, 100 total, recording the average attempts needed to violate, which the authors define as higher is more robust. Phase two applies successful prompts to other compute tiers, 80 total, recording transfer success rate, which the authors define as lower is more robust. Categories are Erotic Content, Illicit Behavior, Extremist Content, and Self Harm, targeting refusal or safety completion policy.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On well-defined tasks, the authors say attacks more often fail; except for Misuse Prompts, think-less, and nerd-sniping.

   Experimental setup

   • Models: o1-preview, o1-mini; vision is o1-v. The attack model for LMP is not named in the paper. Human red teaming reports only o1-preview; vision reports only o1-v.
   • Tasks: addition, multiplication, MATH/MathD, with targets 42, +1, ×7. The 2-digit and 4-digit notations in the figure captions are inconsistent; see the previous question. Also StrongREJECT, Misuse Prompts, Past Misuse Prompts, rule following, AdvSimpleQA, ImageNet-A, Attack-Bard, Attack-Bard-clean.
   • Attacks and resources: many-shot is measured by token count; LMP allows at most 25 per trajectory; soft token is measured by count, steps, or norm; red teaming has 40 people, 5 request levels, and does not sweep resources; images do not sweep resources. No defensive prompting except for images.
   • Metrics: attack success probability; StrongREJECT is [email protected]; vision is category accuracy, with any wrong prediction counted as attack success; red teaming is average number of attempts and transfer success rate.
   • Evaluation and sample sizes: AdvSimpleQA is a neural network, which the authors state has perfect agreement with humans. Free-form jailbreaks are judged manually by experts. The paper does not report the number of prompts, samples per item, or repetitions for most conditions. Gray cells in figures indicate insufficient samples for that inference-compute calculation.
   • Not done: no adversarial training on the studied tasks, and the attack form is not told to the model. The paper does not give absolute units for inference compute.

   Main results

   The complete values in the text that can be checked point by point are mainly in the human red teaming. Most curve endpoints in Figures 1–16 are not written as numbers.

   Requested inference compute level	Average attempts for successful attack	Transfer success rate (n=80)
   1	11.0	36%
   2	16.8	30%
   3	15.6	21%
   4	15.9	20%
   5	22.6	19%

   According to Table 2, the target is o1-preview. The higher the number of attempts and the lower the transfer success rate, the more robust the authors define it to be.

   • Red teaming: the authors state that the lowest level is weakest, the highest level is most resistant to direct attacks, and the top three levels are similar in transfer. The number of attempts does not increase monotonically with level.
   • Figure 1 overview: the horizontal axis is defender inference compute (log scale). The authors state that for unambiguous tasks, increasing inference compute drives down attack success probability; for misuse prompts, LMP often finds a phrasing such that the response does not clearly violate policy. The caption for (e) writes both StrongREJECT misuse prompts and adversarially manipulated images; (f) is labeled Attack-Bard.
   • Math many-shot: the authors state that with fixed attack resources, success rate tends toward 0 as inference compute increases (Figure 2). The ×7 target first fails when compute is very low, then rises before tending toward 0. The authors believe that initially compute is insufficient and the model has not yet caught up with the attacker's pattern, and state that this is consistent with Anil et al. (2024). For o1-preview the authors state it is qualitatively similar (Figure 20).

   Strategy tasks and LMP

   • Many-shot: the authors state that o1-mini and o1-preview are only mildly susceptible to Misuse Prompts. The Figure 5 caption writes that ASR for Misuse Prompts is below 5%, Past Misuse Prompts is at most 25%, and at first glance increasing compute does not help. After inspecting samples, the authors believe the task is ambiguous and test-time compute may not help. Appendix A.2 uses examples to show that grading and "whether it is illegal" can be inconsistent; responses are not quoted here.
   • StrongREJECT suite: the authors state that [email protected] rises with inference compute and is close to monotonic (Figure 7). The paper does not report specific scores.
   • LMP: math results are in Figure 3 (o1-mini) and Figure 8 (o1-preview), which the authors state are qualitatively similar; rule following is in Figures 6 and 9. The point-by-point ASR values in these figures are not given in the text. On misuse prompts, the authors state they see no evidence that increasing compute improves robustness, and the attacker continues to succeed within few attempts; the authors attribute this to inconsistency between the StrongREJECT grader and the defender's policy, rather than a robustness failure (Figure 11).

   Soft token, web injection, and vision

   • Soft token: if attack strength is measured by the number of unconstrained soft tokens, the authors state that success rate quickly approaches 1. During optimization the norm can exceed 1000× the original embedding norm (main text, Figure 21a). After norm constraints, the authors state that more thinking time helps the defender on roughly all tasks; the defender struggles more when the core task is harder and the attack target is easier. The task in Figure 13 is multiplication with the target being the correct answer +1: the authors state that more inference compute improves defense, but sufficiently many soft tokens remain quite effective at the tested compute. The authors state that training in Figure 13b is not very stable, but success does not require selecting checkpoints.
   • AdvSimpleQA: the authors state that after increasing test-time compute, ASR drops to 0 in most settings (Figure 14). The vertical axis is the number of injections in one figure and success rate in another. In the appendix's multi-webpage browsing (number of injections set to 1), the authors state that ASR approaches 0 after compute exceeds a certain threshold (Figure 22). Increasing injections from 1 to 256, the authors state, does not substantially increase ASR (Figure 23); the authors offer one possible explanation: non-browsing messages may be interspersed between the last browsing message and the final output.
   • o1-v: on ImageNet-A, Attack-Bard-clean, and Attack-Bard, the authors state that increasing test-time compute generally improves performance; on Attack-Bard it consistently improves robustness, and it also improves on clean images (Figure 15). The paper does not give accuracy numbers. These images are not adaptively optimized against o1-v.

   Think-less and nerd-sniping

   • Think-less: 64-shot. The Figure 16 caption writes that the blue line (think-less) is always above the orange line (standard many-shot). The authors state that this attack is more effective on o1-mini and weaker on o1-preview. Figure 17 is "compute without the attack / compute with the attack", covering all datasets considered; the authors state that the compute decrease for o1-mini is larger than for o1-preview. Whether the border indicates exceeding 25 percentage points is not given figure by figure in the text, so it is not written here.
   • Whether it is disguised less thinking: the authors state that under most other attacks, the inference compute distribution changes little, and is sometimes longer, e.g., norm-constrained soft tokens. Therefore the authors believe that other settings cannot all be regarded as disguises of think-less.
   • Nerd-sniping: except for the lowest request level, instances whose actual compute is in the top 5% have higher ASR on average than the median for that request level (Figure 18). The authors believe the model may spend inference compute on unproductive loops, so pointwise "thinking longer" does not equal being more robust.

   Other ablations and analyses

   • Training steps for a single soft token are in Figure 12; the text does not give endpoint ASR.
   • The steps in Figure 13a were selected by performance on a small validation set; a supplementary figure with unconstrained count is one of the 9 math settings (Figure 21b), with no endpoint numbers.
   • For the 9 cells of math many-shot / LMP (3 tasks × 3 targets), the text gives only qualitative patterns, not per-cell ASR.
   • Red teaming phase one has 20 items per level, 100 in total; the four content categories are in the previous question.
   • Footnote: in some cases attack success probability first rises then falls.
   • Appendix A.6: in one trajectory with unconstrained soft tokens, the model finishes the problem but still ends with the attacker's answer; with norm constraints, the authors state that the full chain of thought cannot be preserved. Attack sentences are not quoted.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors say they only touch specification application and do not answer whether success rates go to zero everywhere with compute.

   Limitations and future directions noted by the authors

   • Only one of four prerequisites is addressed (Section 1.2): the authors list conditions needed for safer responses: the ability to decompose context into distinct components; the existence of a specification delineating allowed and prohibited behavior; the model knowing these specifications; and the model applying the specification to concrete instances, especially out-of-distribution instances caused by an adversary. The authors say this paper only addresses item 4; items 1–3 are out of scope, and item 4 does not yet have a complete, foolproof solution.
   • Specification and compliance separated (Section 1.2): the authors say this paper evaluates whether a model with inference compute can execute a specification, leaving the definition of specifications to other research; and they focus only on cases where the specification is unambiguous. Writing specifications without loopholes and covering edge cases is, in the authors' words, a hard problem requiring dedicated research.
   • Scope (Section 4): the authors say they explored only a limited set of tasks and a limited range of compute scaling, and the work remains preliminary.
   • Whether it goes to zero (Section 4): whether attacker success rates go to zero with inference compute in all settings remains open, the authors say. When exploiting policy ambiguity or loopholes, scaling compute does not appear to help.
   • New attack surface (Section 4): using inference compute as a safety measure opens up think-less; another is nerd-sniping, where the model spends its budget on tasks unrelated to the policy. The authors say LMP found this distraction attack in some cases, where inference compute can be unusually large.
   • Follow-up attacks (Section 3.6, Section 4, Appendix A.7): stronger image adversarial attacks; multimodal token gradient methods more realistic than unconstrained soft tokens; stronger browsing-agent prompt injection.

   Experimental coverage

   • Models: experiments ran on o1-preview, o1-mini, and o1-v. Human red-teaming numbers come only from o1-preview (Table 2); vision accuracy comes only from o1-v (Figure 15).
   • Tasks and attacks: Table 1 and Section 2 list addition, multiplication, MATH, target 42, +1, ×7; StrongREJECT (35 jailbreaks, 60 prompts), Misuse Prompts, Past Misuse Prompts, rule following, AdvSimpleQA, ImageNet-A, Attack-Bard, and clean. LMP has a cap of 25 turns per trajectory; think-less uses 64-shot.
   • Training and specifications: the authors did no adversarial training on the studied tasks; specifications for policy tasks are not written into the context (Section 2). Image experiments write the class name into the prompt (Section 3.6).
   • Not reported in the paper: number of prompts for most conditions, number of samples per prompt, number of repetitions, sampling temperature, absolute units of inference compute, and the name of the LMP attacker model. [email protected] and o1-v accuracy have no pointwise numbers.
   • Explicitly stated judgments: for AdvSimpleQA, the authors say human and neural-network scoring agreement is perfect, without giving the sample size for that comparison; red-team transfer is n=80 (Table 2). The perturbations for ImageNet-A and Attack-Bard are not optimized against these models (Section 2.2, 3.6).

   Collapse this answer

6. 06Summarize the main content of the paper

   Increasing inference compute reduces attack success rates on several well-defined tasks, but cannot fill in specifications and also introduces new attacks.

   The authors measure how the adversarial robustness of the o1 series changes with inference-time compute, without adversarial training against the tested attacks.

   • Problem: Agents read untrusted input. Adversarial training requires knowing the perturbation set T in advance, and the authors consider training on known attacks whack-a-mole. In the jailbreak results they cite, pretraining compute correlates negatively with robustness, so they instead look at inference-time compute.
   • Approach: Victim models are o1-preview, o1-mini, and o1-v for vision. The adversary controls only part of the prompt, or rewrites inputs in misuse tasks. Attacks include many-shot, the StrongREJECT suite, black-box LMP with up to 25 attempts, white-box soft tokens, a 40-person red team, and ImageNet-A / Attack-Bard. There are also think-less (suppressing actual reasoning) and nerd-sniping (spending compute on unproductive reasoning). Policy specifications are not written into the context.
   • Clear tasks: The authors state that with fixed attack resources, the success probability of math many-shot tends to 0 with inference compute, but the ×7 target first rises then falls. On AdvSimpleQA, the authors state ASR drops to 0 in most settings (Figure 14). For StrongREJECT [email protected], the authors state it is nearly monotonically increasing (Figure 7); specific scores are not given.
   • Red team with numbers: On o1-preview, the lowest request tier averages 11.0 attempts with 36% transfer success; the highest tier is 22.6 attempts and 19% (Table 2, transfer n=80). The authors state the lowest tier is weakest and the top three tiers transfer similarly.
   • Exceptions: For Misuse Prompts many-shot, the figure caption shows ASR below 5%; the past-tense version reaches 25% at most, which the authors attribute to task ambiguity. On LMP misuse, the authors state they see no robustness from compute and attribute it to grader-policy mismatch. Think-less is more effective on o1-mini. Instances whose actual compute is in the top 5% have higher average ASR.
   • Authors' conclusions: Putting scale into inference-time compute may yield scale benefits in adversarial settings, and compute can be temporarily increased in high-risk situations. The authors also state this cannot compensate for missing specifications, nor guarantee that an adversary cannot force the model to think less or exhaust budget on reasoning unrelated to policy.

   Collapse this answer

Related papers