Skip to content

BrainwashedLLM: Jailbreaking via Automated Discovery of Adversarial Chains of Thought

BrainwashedLLM: Jailbreaking via Automated Adversarial Chain of Thoughts Discovery

arXiv: Jailbreaks, Prompt Injection, Poisoning & Defenses· arXiv:2611.04717· Masoud Poorghaffar Aghdam, Göktuğ Serdar Yıldırım, Sinem SavDepartment of Computer Engineering, Bilkent UniversityIndexed by this site · 16 days ago· Originally publishedSelected AttacksSelectedAttention39

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors propose Brainwashed LLM, which uses GRPO to fine-tune a small model to generate malicious reasoning traces and injects them into a large model, achieving ASR@5 of 0.74–0.98 on four LRMs.

Threat modelWeak-to-strong blind attack: the attacker has full white-box access to the smaller source model (gradients, parameters) and only black-box reasoning access to the larger target model; throughout attack construction it never queries the target, never observes its outputs, and obtains no feedback. The target is an LRM whose reasoning trajectory is externally visible; models that hide the CoT (such as GPT o-series) are outside the threat model.

Questions
Large reasoning models (LRMs) explicitly output a chain of thought (CoT), and the authors consider this intermediate reasoning surface itself to be an attack vector: safety alignment is more vulnerable to malicious reasoning embedded in context than to direct imperative prompts, while existing automated attacks only optimize the final output tokens, not the reasoning traces.
Methods
Proposes Brainwashed LLM: using GRPO with multi-objective rule rewards (refusal, safety advice, length) to fine-tune a small white-box source model in two stages so that it generates malicious CoT without refusal; then injects this trajectory as a prefix into a larger black-box target LRM, letting it continue writing harmful content, without querying the target throughout.
Experiments and results
On the four target LRMs, reasoning injection raises ASR@5 from 0.20–0.48 for normal prompts to 0.74–0.98 (Figure 4a); compared with Mousetrap and H-CoT, which require online interaction, the gap is at most 16 percentage points (Table I, baselines use their own datasets).
Limitations and future work
The authors note: only textual reasoning trajectories are explored, multimodal or tool-using agents are not covered; transfer results are limited to targets of about 30B parameters, and 70B+ models were not evaluated due to hardware constraints; no empirical comparison was made with other ways of producing malicious trajectories, such as unaligned models or handwritten trajectories (Appendix H).
Experimental setupQwen3 8B Instruct、DeepSeek R1 Distill Qwen2.5 14B et al. · Harmful Behaviors、JailbreakBench et al. · ASR@5、ASR@1
Evaluated models
Qwen3 8B InstructDeepSeek R1 Distill Qwen2.5 14BQwen3-32BDeepSeek-R1-Distill-Qwen-32BMagistral-Small-2506Gemma-4-31B
Benchmarks
Harmful BehaviorsJailbreakBenchTrotterStrong(Mousetrap 原论文数据集)H-CoT 原论文数据集
Metrics
ASR@5ASR@1
AI summaryand selection rationaleResearchers at Bilkent University in Turkey propose the BrainwashedLLM jailbreak framework, which uses GRPO reinforcement learning to fine-tune small white-box models such as Qwen3-8B to generate adversarial chains of thought; the reward is computed entirely based on the source model's own reasoning chain, and the entire attack construction never queries the target or obtains gradients or output feedback. The injected reasoning chain is treated by larger black-box reasoning models as their own prior reasoning and continued, producing harmful content. On four models—Qwen3-32B, DeepSeek-R1-Distill-Qwen-32B, Magistral-Small-2506, and Gemma-4-31B—ASR@5 reaches 0.74 to 0.98, a gap of at most 16 percentage points from the Mousetrap and H-CoT baselines that require online interaction. The authors state that this is the first framework that requires no target feedback and directly optimizes the chain of thought as the attack surface via RL, and explain that the attack applies only to models whose reasoning process is externally visible; GPT o series models that hide their chain of thought are outside its threat model, and they recommend architecture-level defenses such as isolating the reasoning buffer.

Researchers at Bilkent University in Turkey propose the BrainwashedLLM jailbreak framework, which uses GRPO reinforcement learning to fine-tune small white-box models such as Qwen3-8B to generate adversarial chains of thought; the reward is computed entirely based on the source model's own reasoning chain, and the entire attack construction never queries the target or obtains gradients or output feedback. The injected reasoning chain is treated by larger black-box reasoning models as their own prior reasoning and continued, producing harmful content. On four models—Qwen3-32B, DeepSeek-R1-Distill-Qwen-32B, Magistral-Small-2506, and Gemma-4-31B—ASR@5 reaches 0.74 to 0.98, a gap of at most 16 percentage points from the Mousetrap and H-CoT baselines that require online interaction. The authors state that this is the first framework that requires no target feedback and directly optimizes the chain of thought as the attack surface via RL, and explain that the attack applies only to models whose reasoning process is externally visible; GPT o series models that hide their chain of thought are outside its threat model, and they recommend architecture-level defenses such as isolating the reasoning buffer.

Why it was selectedDuring the construction phase, the attack relies entirely on neither queries to nor feedback from the target model, and can zero-shot jailbreak four larger open-source reasoning models using only the reward signal from the small model's own reasoning chain.

Related events and news

1 events · 1 news mentions

News that mentions this paper

大模型安全周报:智能体安全边界在签名、授权与上下文

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors argue that the LRM's explicit reasoning trajectory is an attack vector, and propose an RL framework that automatically generates reasoning-based jailbreaks without target feedback.

    Core question: whether the chain of thought (CoT) exposed by large reasoning models (LRMs) can itself be used automatically as an attack surface to achieve cross-model jailbreaks without querying the target model.

    • Setting and importance: LRMs perform multi-step reasoning through an explicit CoT, which the authors argue changes how information is represented and exposed at inference time; existing safety evaluations rely on static benchmarks and cannot cover adaptive attackers (Introduction).
    • Shortcomings of existing methods: the authors state that existing automated attacks (such as LLMStinger, Mousetrap, H-CoT) all rely on online interaction with the target to optimize prompts, and the optimization target is the final output tokens or the prompt surface rather than the intermediate reasoning trajectory (Related Work, Section IV-D).
    • Core observation/hypothesis: LRMs are trained to keep the reasoning trajectory logically consistent with the final answer; the authors hypothesize that after a malicious reasoning trajectory is injected as a context prefix, the target model will treat it as its own prior reasoning and continue writing harmful content (Section III-E).
    • Proposed in the paper: Brainwashed LLM—fine-tune a small white-box source model with GRPO plus multi-objective rule rewards to generate malicious CoT, then inject it zero-shot into larger black-box target LRMs; the authors describe this as a framework that uses RL to directly optimize the LRM's CoT as an attack surface with no target feedback, and demonstrate zero-shot weak-to-strong transfer of reasoning trajectories (Abstract, Introduction).
    • Threat model (Section III-A):
      • Goal: for a malicious query q, generate a reasoning trajectory τ that, as an input prefix, forces the target model Mtarget to continue into a harmful answer y.
      • Knowledge: full white-box access to the smaller source model Msource (gradients, parameters θ); only black-box reasoning access to the target model.
      • Capability: never queries the target, never observes its outputs, and never fine-tunes the target throughout attack construction; can only input text context to the target.
      • Victim systems: LRMs whose reasoning trajectories are externally visible; the authors explicitly place models that hide the CoT (such as GPT o-series) outside the threat model because prefix injection is not possible.
  2. What related research exists?

    Related work falls into three groups: jailbreaks that exploit the reasoning process, methods that treat jailbreaking as a learning problem, and reasoning-centric attack baselines.

    Jailbreaks exploiting reasoning and conversational processes

    • Multi-turn/conversational jailbreaks (Crescendo [20], Chain of Attack [22], etc. [14], [21], [23]): spread malicious intent across multiple seemingly benign turns, exploiting conversational memory and coherence to elicit harmful behavior; persuasion and role-play attacks [24], [25] guide the model across boundaries through natural dialogue.
    • Attacks on LRM reasoning [21], [26]–[28]: hide malicious intent in complex problem-solving tasks to bypass safety alignment; there are also template and optimization techniques [29] and attacks via excessive/insufficient/overflowing reasoning steps [30]–[33]. The paper notes that these attacks are hand-designed or require interaction with the target, and do not optimize reasoning trajectories offline.

    Treating jailbreaking as a learning problem

    • Don't Say No [34]: reduces the model's ability to refuse harmful requests by suppressing refusal tendencies, independent of model architecture; this paper's refusal pattern word list is partly taken from that work.
    • DRL-guided search [35] and xJailbreak [36]: model jailbreak generation as a sequential decision problem, using deep RL or representation-space guidance to discover attack strategies.
    • LLMStinger [18]: fine-tunes an attack model with PPO plus a judge LLM and a string-similarity checker. The paper notes that these RL methods attack the input prompt surface of standard LLMs, depend on target output feedback at every training step, and transfer prompt-level artifacts such as suffixes and conversation templates, whereas this paper optimizes and transfers the reasoning trajectory itself.

    Reasoning-centric attack baselines

    • Mousetrap [14]: iteratively rewrites prompts based on target responses; H-CoT [15]: first queries the target with a weakened malicious prompt to collect its CoT trajectory, then imitates it. Both require direct interaction with the target throughout attack development; this paper compares ASR@5 with them in Table I.
    • AutoRAN [38] (described by the authors as closely related parallel work): also performs weak-to-strong jailbreaking of LRMs, but uses reasoning signals leaked in the target's refusals to iteratively optimize prompts, requiring online interaction and acting on the prompt surface.

    Baselines and benchmarks: the comparison baselines are Mousetrap and H-CoT (each using its original paper's dataset); evaluation uses Harmful Behaviors and JailbreakBench.

    The authors position this paper as: under a threat model with weaker assumptions at the attack construction stage (blind, offline), using RL to optimize the source model's reasoning trajectories and transfer them zero-shot to larger black-box LRMs, with ASR within 16 percentage points of baselines that require online interaction (end of Related Work).

  3. How does the paper address this problem?

    Two stages: fine-tune a small source model with GRPO plus multi-objective rule rewards to generate malicious CoT, then inject it as a prefix into a large target model to continue.

    The overall idea is to model attack generation as an RL optimization problem on the source model, with rewards computed entirely from the source model's own reasoning trajectories; once trained, the malicious CoT is injected zero-shot into the target model. Brainwashed LLM has two stages: source model fine-tuning (Phase 1) and malicious reasoning injection (Phase 2).

    Problem setting and formalization

    • The source model Msource acts as policy πθ, generating outputs o (the concatenation of reasoning trajectory τ and final answer y) for q in the malicious prompt dataset Dmal.
    • The optimization objective is θ∗ = argmaxθ 𝔼q∼ Dmal, o∼ πθ(·|q) [r(q, o)], where the composite reward r suppresses refusal and safety-advice language in the reasoning trajectory and encourages substantive reasoning length.

    Multi-objective rule rewards (Section III-D)

    • Refusal reward: fixed regular expressions match refusal patterns in the reasoning segment (before the special reasoning-end token); no refusal scores 1, otherwise the score is based on the ratio of the refusal position pref to the segment length |t|, with earlier refusals penalized more; a missing end marker scores −1. The refusal word list combines phrases from [34] and phrases the authors observed across models.
    • Safety-advice reward: count the hits H of predefined safety-advice phrases in the reasoning segment, normalize by α = min(1, H/Lcap), and take rsafe(o) = 1 − sin(απ/2), decreasing smoothly from 1 to 0 as safety advice increases.
    • Length scaler reward: uses the cosine shaping function of [41], depending only on the CoT token length Lgen and the maximum length Lmax, with boundary rewards r0 and rL; the authors state that a low r0 is set to prevent the model from generating empty reasoning to evade the penalty (reward hacking), while also penalizing excessive length to avoid truncation.
    • Composite reward: r(q, o) = rrefusal · rscaler + rsafe; the authors state that the scaler is multiplied only onto the refusal term because they found it was easily gamed when unweighted.

    GRPO optimization and two-stage training (Section III-C, Appendix A/B)

    • Reasons for choosing GRPO (given by the authors): no paired preference data, ruling out DPO; GRPO replaces PPO's critic network with within-group relative advantages, reducing memory so that low-resource attackers can use it.
    • Key hyperparameters (Appendix B): LoRA (4-bit QLoRA, rank and alpha both 64) fine-tuning for 100 epochs, batch size 48, learning rate 1e−4, KL penalty β = 0.02, group size G = 8, maximum sequence length 4096, Lcap = 16, Lmax set to two-thirds of the maximum sequence length.
    • Two stages: Stage I (Warmup) uses only the refusal reward plus the scaler reward, first breaking immediate refusal; Stage II adds the safety-advice penalty to obtain the full composite reward. The authors state that using the full reward from the start leads to unstable training and non-convergence.

    Malicious reasoning injection (Section III-E)

    • The converged source model generates a reasoning trajectory τweak for each malicious prompt, which is concatenated with q (according to each target model's chat template) as the target model's input, letting the target "continue" the reasoning to obtain ytarget.
    • The authors' design intuition: LRMs are trained to keep reasoning and answer logically consistent, so they will treat the injected trajectory as their own prior reasoning and complete it as a fixed premise.

    Success determination (Section IV-B)

    • Dual-judge agreement: HarmBench Judge (a classifier based on Mistral-7b) and NLI Judge (BART-Large-MNLI) each make a judgment; success requires both to judge the output harmful, and disagreements default to safe.
    • Metric ASR@N: a prompt's attack succeeds if at least one of N samples is judged harmful by both judges; in the injection experiments, the target's complete continuation is scored.
  4. What experiments did the paper conduct?

    Two source models jailbroken, reasoning injection on four target models, compared with Mousetrap/H-CoT; after injection ASR@5 reaches 0.74–0.98.

    Experimental Setup

    • Source models (fine-tuned with GRPO): Qwen3 8B Instruct and DeepSeek R1 Distill Qwen2.5 14B (also referred to in the main text as Qwen3-8b and DeepSeek-R1-Distill-Qwen-14B), using the Unsloth library.
    • Target models (injection victims, 4 in total): Qwen3-32B, DeepSeek-R1-Distill-Qwen-32B, Magistral-Small-2506, Gemma-4-31B, all locally run open-weight models (Appendix B).
    • Data: 50 prompts randomly drawn from the Harmful Behaviors dataset for training (seed 42; the authors state that training on the full set does not converge); JailbreakBench for generalization evaluation, the paper does not state the size of the subset used, and notes that other subsets or seeds were not tested.
    • Sampling: temperature T ∈ {0, 0.5, 1.0, 1.5, 2.0}, with N = 5 responses sampled per temperature (at T = 0, deterministic decoding generates only 1 response, so ASR@5 equals ASR@1).
    • Judging: the intersection of two judges, HarmBench Judge and NLI Judge; disagreements default to safe.
    • Baselines: Mousetrap and H-CoT, each using its own original paper's dataset (Mousetrap uses TrotterStrong); all three are scored with the same evaluation pipeline; the authors note that the prompt sets differ, so the gaps reflect both the attack and the dataset.

    Main Results

    Comparison of ASR@5 between reasoning injection (Broken CoT, trajectories from the Qwen3-8B source model) and normal prompts (Normal Prompt) (Figure 4a), and comparison with baselines (Table I):

    Swipe horizontally to read the table

    Target modelNormal Prompt ASR@5 (Fig. 4a)Broken CoT ASR@5 (Fig. 4a)Mousetrap (Table I)H-CoT (Table I)
    Qwen3-32B0.200.881.000.96
    DeepSeek-R1-Distill-Qwen-32B0.480.901.000.98
    Magistral-Small-25060.220.980.980.98
    Gemma-4-31B0.240.740.800.90
    • The authors state that after injection ASR@5 rises on all four targets, with a gap of at most 16 percentage points versus Mousetrap and H-CoT, a gap of zero on Magistral-Small-2506, and that this paper's method never queries the target throughout attack construction.
    • Figure 4b reports ASR@1; the authors state that the effect still holds in the single-shot setting (the main text does not give specific ASR@1 values for each target).
    • Authors' interpretation: safety guardrails designed for imperative prompts are markedly weaker against malicious intent injected in reasoning trajectories, and this holds on all four targets.

    Source Model Jailbreak Effectiveness and Generalization

    • Jailbreaking source models: Figure 3 reports ASR@5 at each temperature before and after jailbreaking for the two source models. The authors state that the two jailbroken models have ASR@5 of at least 0.86 at all non-zero temperatures, whereas base Qwen3-8b is below 0.20 and DeepSeek-R1-Distill-Qwen-14B is between 0.44 and 0.80; at T = 0.5, ASR@1 is at least 0.73 (Appendix C).
    • Gap between ASR@1 and ASR@5: for Qwen3-8b on Harmful Behaviors at T = 0.5, ASR@1 is 0.74 and ASR@5 is 0.94; for DeepSeek on JailbreakBench at T = 2.0, the two metrics differ by about 21%. The authors argue that single-shot ASR@1 underestimates true vulnerability and that refusals are stochastic.
    • Generalization: on the independent JailbreakBench, ASR@5 is 0.95 for Qwen3-8b and 0.90 for DeepSeek; the authors state that the attack does not overfit to the training prompts.

    Ablations and Representation Analysis

    • Reward ablation (Appendix D, Figure 8): on Qwen3-8B, the model using only the Stage 1 reward drops in ASR@1 from 0.73 to 0.56 at high temperature (training set); the full two-stage reward narrows the gap between ASR@1 and ASR@5. The authors state that the Stage 2 safety-advice penalty improves single-shot reliability.
    • Representation analysis (Appendix E, Figure 9): 50 CoTs each from base and jailbroken Qwen3-8B are embedded with Qwen3-Embedding-8B and then t-SNE is applied (perplexity 15, seed 42). In the figure, base outputs concentrate in the left and upper-left regions, jailbroken outputs occupy the right and lower-right, with limited overlap; the authors state that GRPO fine-tuning moves the source model's output distribution to a different region of the semantic space.

    Other Ablations and Analyses

    • Training data size: training on the full Harmful Behaviors set does not converge, so 50 random prompts are used instead (seed 42, Section IV-A).
    • Compute cost (Appendix B): a single NVIDIA L40S 48GB, about 48–72 hours per model for adversarial fine-tuning, and about 8–12 hours for the evaluation pipeline.
    • Qualitative examples (Appendix G): side-by-side examples of DeepSeek-R1-Distill-Qwen-32B and Qwen3-32B outputs under normal refusal versus after injection; the authors state that they demonstrate weak-to-strong transfer.
    • Negative result: the authors mention that architectures trained to refuse immediately within the CoT, such as Phi-4-reasoning, cause sparse rewards, leaving iterative RL without intermediate feedback (Appendix H, no experimental numbers for that model are given).
  5. What are some points that can be further explored?

    The authors list limitations on modality, model scale, and reward generalization; experiments cover four ~30B targets and 50 training prompts.

    Limitations and future directions noted by the authors

    • The method currently relies on textual reasoning trajectories and has not yet explored effects on multimodal reasoning models or agents with tool-use capabilities (Appendix H Limitations).
    • The rule rewards are effective for general harm categories, but highly domain-specific attacks may still require domain knowledge to construct effective reasoning chains (Appendix H Limitations).
    • Transfer results are limited to the largest target evaluated (about 30B parameters); 70B+ frontier models could not be evaluated due to hardware constraints (Appendix H Limitations).
    • No empirical comparison was made between GRPO and other ways of producing adversarial trajectories (such as unaligned models or hand-written trajectories); the authors state that closing these gaps is a natural future direction (Appendix H Limitations).
    • For architectures that refuse immediately within the CoT, such as Phi-4-reasoning, the reward signal is sparse; the authors plan to address this in the future by pre-fine-tuning the source model to introduce variance and by relaxing the immediate-refusal boundary (Appendix H).
    • The authors propose potential defense directions (circuit breakers that monitor the CoT in real time, limiting user control over the reasoning buffer), and state that evaluating these and existing jailbreak defenses against this paper's attack is left to future work (Appendix H).

    Scope of experiments

    • The injected target models are Qwen3-32B, DeepSeek-R1-Distill-Qwen-32B, Magistral-Small-2506, and Gemma-4-31B, 4 in total, all locally run open-weight models (Section IV-B, Appendix B).
    • The source models are Qwen3 8B Instruct and DeepSeek R1 Distill Qwen2.5 14B, 2 in total; the trajectories used in the injection experiments come from the Qwen3-8B source model (Section IV-C).
    • Training data are 50 random prompts from Harmful Behaviors (seed 42); generalization evaluation uses JailbreakBench, and other subsets or seeds were not tested (Section IV-A).
    • The baseline comparison covers only two reasoning-centric attacks, Mousetrap and H-CoT, each using its original paper's dataset (Table I, Appendix F).
    • The paper does not report agreement between judge verdicts and human annotation, nor the specific number of prompts per target in the injection experiments.
  6. Summarize the main content of the paper

    GRPO fine-tunes a small model to produce malicious CoT and prefix-injects it into a large model, achieving ASR@5 of 0.74–0.98 on four LRMs without target feedback.

    A jailbreak attack paper targeting large reasoning models (LRMs), whose core claim is that the exposed chain of thought is itself an attack surface.

    • Problem: the authors argue that LRM safety alignment is more fragile against malicious reasoning embedded in context than against direct imperative prompts, while existing automated attacks require online interaction with the target and optimize only the prompt or the final output tokens.
    • Method: Brainwashed LLM uses GRPO plus multi-objective rule rewards (refusal position, safety-advice density, length shaping) to fine-tune a small white-box source model in two stages so that it generates refusal-free reasoning trajectories for arbitrary harmful prompts; the trajectories are then injected as a prefix into larger black-box target LRMs for them to continue. Attack construction never queries the target or observes its outputs.
    • Source model results: after jailbreaking, Qwen3-8b and DeepSeek-R1-Distill-Qwen-14B have ASR@5 of at least 0.86 at non-zero temperatures, and 0.95 and 0.90 respectively on the independent JailbreakBench (Figure 3, Section IV-C).
    • Injection results: on Qwen3-32B, DeepSeek-R1-Distill-Qwen-32B, Magistral-Small-2506, and Gemma-4-31B, reasoning injection raises ASR@5 from 0.20–0.48 for normal prompts to 0.74–0.98 (Figure 4a).
    • Comparison with baselines: compared with Mousetrap and H-CoT, which require online interaction, the ASR@5 gap is at most 16 percentage points, and is level on Magistral-Small-2506 (Table I, baselines use their own datasets).
    • Authors' conclusion: the LRM's training commitment to logical consistency leads it to treat the injected malicious reasoning as a given premise; the authors argue this exposes the reasoning buffer as a vulnerable point, and suggest researching defenses such as real-time monitoring of reasoning trajectories and isolation of the reasoning buffer; the authors describe this as a framework that uses RL to directly optimize the LRM's CoT as an attack surface with no target feedback.

Related papers

Show 3 more papers
Read originalarxiv.org