Skip to content

ShallowDefend: a multi-turn defense method leveraging shallow safety alignment

ShallowDefend: Harnessing Shallow Safety Alignment for Multi-Turn Defender

arXiv· arXiv:2610.40627· Qiyao Ma, Xiaodong Liu, Weiwei Yang, Mingqian Feng, Chengquan Guo, Jialin Song, Carolyn Buractaon, Zhe Zhao, Junshan Zhang, Sandeep Atluri, Christopher White, Jianfeng GaoUniversity of California, Davis; Microsoft ResearchIndexed by this site · 21 days ago· Originally publishedDefenses

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

The authors propose a multi-turn defense framework, Assess → Decide → Respond, using RL to train the model to first commit to a parseable safety decision before replying; the authors claim an average SAFE rate of 90.4%–93.4% on Llama3.1-8B and Qwen3-8B, outperforming baselines trained on the same data.

Threat modelThe threat model is scripted multi-turn attacks, in which the attacker spreads malicious intent across multiple seemingly harmless user turns and delivers the payload in the final turn; the attack controls only user turns, is statically replayed, and cannot write to the assistant channel; the authors acknowledge adaptive adversaries as an unresolved limitation. The victim systems are the Llama3.1-8B-Instruct and Qwen3-8B chat models.

Questions
Safety-aligned models can reliably refuse single-turn harmful requests, but the same request spread across multiple turns still often succeeds. The authors believe the root cause is that alignment is shallow: safe behavior is concentrated in the first few tokens of the response, existing multi-turn defenses only supervise the surface form of responses, and the model never learns an explicit safety decision.
Methods
Proposes Assess → Decide → Respond: each turn, first assess user intent, then commit to a parseable decision label (COMPLY/REFUSE, with SAFE-REDIRECT added in the external-reward version), and then generate a response accordingly; GRPO applies reward only at the final turn, and the self-verification reward uses rules to compare the committed label against known properties of the session, requiring no judge model.
Experiments and results
On Llama3.1-8B and Qwen3-8B across five safety benchmarks, the authors report that the average SAFE rate of the two reward variants (90.4%–93.4%) exceeds all baselines trained on the same data, and remains ahead under six judges; utility is essentially unchanged, at the cost of a higher over-refusal rate (e.g., XSB for Qwen3-8B self reward rises from 15.6% to 56.8%).
Limitations and future work
The authors acknowledge that the threat model covers only statically replayed scripted attacks, and that adaptive adversaries are an open limitation; the ordering of the two reward variants reverses with the backbone model, and the authors state that the experiments did not isolate the cause; inter-judge agreement is only moderate, and a single judge's success rate should not be treated as ground truth.
Experimental setupLlama3.1-8B-Instruct、Qwen3-8B · AdvBench、HarmBench et al. · SAFE %、over-refusal (XSTest/XSB) et al.
Evaluated models
Llama3.1-8B-InstructQwen3-8B
Benchmarks
AdvBenchHarmBenchMultiBreakCoSafeMHJMMLUHellaSwagWinograndeGSM8KTruthfulQAIFEvalXSTestXSBMT-BenchUltraChat-200k
Metrics
SAFE %over-refusal (XSTest/XSB)MT-BenchUtility Avg.decision-flip rate
AI summaryThe researchers propose the ShallowDefend framework, in which the model first produces an explicit, parseable safety decision before generating a response, achieving the strongest multi-turn safety performance in controlled comparisons across two model families and fourteen benchmarks. The work targets the problem of shallow safety alignment, where safety behavior is concentrated in the first few tokens of model output, and the same harmful request distributed across multiple turns often still succeeds. The framework operates in three steps—evaluation, decision, response—with the decision commitment trained via reinforcement learning on the payload turn; a rule-based self-verification reward compares the commitment label against known properties of the dialogue, requiring neither a judge model nor human annotation, and the same format also supports an LLM-as-judge graded decision space and a safety-steering reward. The defense maintains a safety lead while preserving utility, at a moderate cost to helpfulness. Per-token KL analysis and intervention probes show that the training effect is concentrated in the brief thinking segment that determines the response trajectory, where the committed decision is a causal control point, and the shallowness exploited by attackers instead becomes a defense mechanism.

The researchers propose the ShallowDefend framework, in which the model first produces an explicit, parseable safety decision before generating a response, achieving the strongest multi-turn safety performance in controlled comparisons across two model families and fourteen benchmarks. The work targets the problem of shallow safety alignment, where safety behavior is concentrated in the first few tokens of model output, and the same harmful request distributed across multiple turns often still succeeds. The framework operates in three steps—evaluation, decision, response—with the decision commitment trained via reinforcement learning on the payload turn; a rule-based self-verification reward compares the commitment label against known properties of the dialogue, requiring neither a judge model nor human annotation, and the same format also supports an LLM-as-judge graded decision space and a safety-steering reward. The defense maintains a safety lead while preserving utility, at a moderate cost to helpfulness. Per-token KL analysis and intervention probes show that the training effect is concentrated in the brief thinking segment that determines the response trajectory, where the committed decision is a causal control point, and the shallowness exploited by attackers instead becomes a defense mechanism.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Multi-turn jailbreak success rates are far higher than single-turn; the authors believe the root cause is shallow alignment, and propose turning the shallow position into an explicit decision point that defenders occupy first.

    Core problem: safety-aligned models can reliably refuse single-turn harmful requests, but the same request spread across multiple turns still often succeeds, and existing defenses do not address this gap.

    • Problem setting: multi-turn attackers spread malicious intent across turns, so each message is harmless in isolation and the payload is only delivered at the payload turn; the authors say human red teams can still defeat defenses reported to have near-perfect single-turn robustness.
    • Shortcomings of existing methods: the authors argue that training-based defenses (SFT, preference optimization, RL with judge reward) all rely on an oracle (harmfulness classifier, preference annotation or guard model) to supervise the surface form of responses; judges disagree on a substantial portion of samples (§4.2), their errors are brought into training, and the model never learns an explicit, rule-checkable safety decision.
    • Core observation: alignment is shallow—the distribution difference between aligned models and the base model appears almost only in the first few output tokens (Qi et al., 2024a), so prefilling an affirmative prefix can break refusal. The authors read this as a map rather than a defect: since a short span of tokens determines the trajectory of the whole response, defenders can occupy it first.
    • What the paper proposes: an Assess → Decide → Respond framework, in which the defender first assesses intent each turn, commits to a parseable safety decision, and then responds accordingly; RL trains this commitment only at the payload turn, and the reward can be a rule comparing against labels (self-verification) or a judge model.
    • Threat model:
      • Attacker goal: through multi-turn dialogue, make the model substantively satisfy a harmful request in the final turn.
      • Knowledge and capability: scripted multi-turn attacks, static replay, controlling only user turns.
      • Victim systems: Llama3.1-8B-Instruct and Qwen3-8B chat models.
      • Boundaries: the authors explicitly state that adaptive adversaries are an open limitation; attackers able to write to the assistant channel are outside the threat model (§A.2).
  2. What related research exists?

    Related work is grouped into three: multi-turn jailbreaks, multi-turn defenses, and shallow alignment; the authors use a controlled comparison on the same data to distinguish this paper from training-based defenses.

    Multi-turn jailbreaks

    • Escalating attacks such as Crescendo (Russinovich et al., 2024) split a harmful goal into requests that are harmless turn by turn; Ren et al. (2025) belongs to this category as well.
    • Persuasion and rapport-building strategies (Zeng et al., 2024); human red teaming (Li et al., 2024)—the authors say its effect is clearly stronger than automated attacks.
    • The authors point out that such attacks never directly make a harmful request, so single-turn defenses and input filtering have almost no effect.

    Multi-turn defenses

    • Training-based methods that supervise responses: SFT and preference optimization (Jiang et al., 2024; Yan et al., 2025), RL with a harmfulness classifier or guard model as reward (Ding et al., 2025; Guo et al., 2025). The authors criticize this design for bringing judge errors into training, entangling training reward with evaluation metrics, and opening room for reward speculation.
    • Representation-level methods: separating harmful and harmless directions in activation space (Zou et al., 2024; X-Boundary of Lu et al., 2025).
    • Inference-time intervention: having the model analyze user intent before answering (Zhang et al., 2025b). The authors say it is closest to this paper's assessment step, but note that it benefits only from deliberation, whereas this paper's ablation shows limited defense recovery without a committed decision (§3.3).

    Shallow alignment

    • Qi et al. (2024a) propose that aligned behavior is concentrated in the first few output tokens; prefilling an affirmative prefix (Andriushchenko et al., 2025; Vega et al., 2023), exploitation of decoding parameters (Huang et al., 2024) and fine-tuning perturbation (Qi et al., 2024b) all exploit this point to break refusal.
    • An existing remedy is to deepen alignment, for example by training with data augmented from recovery from harmful prefixes (Qi et al., 2024a; Zhang et al., 2025a).

    Baselines and benchmarks: the baselines are SFT, DPO (Rafailov et al., 2023), SDGO, MTSA, MUSE-D and X-Boundary, all trained on the same data as this paper; the authors excluded defenses whose contribution is mainly a dataset rather than a method (Ren et al., 2025; Rahman et al., 2025). The safety benchmarks are AdvBench, HarmBench, MultiBreak, CoSafe and MHJ.

    Authors' positioning: this paper's reward reads only the decision the model itself commits to and compares it against the session's known label, requires no model in the loop, and gradients need not reach the response itself—the authors say this distinguishes it from the three categories of work above.

  3. How does the paper address this problem?

    Each turn is generated as Assess → Decide → Respond, and GRPO is used to train only the committed decision label in the final turn, with rewards being rules or a judge.

    Overall idea: turn the trajectory-determining position left by shallow alignment into an explicit commitment that defenders occupy first—each turn, first assess intent, commit to a parseable decision, and then respond accordingly, with RL training this commitment only at the payload turn. The framework is called Assess → Decide → Respond.

    Output format and decision space

    • The system prompt requires three segments per turn: Assessment (the user's true intent and potential harm), Decision (COMPLY / REFUSE, with SAFE-REDIRECT added in the external-reward version), and Response (a response consistent with the decision); the original text of the two system prompts is in Appendix D (Figure 8).
    • Denote the turn-t output as yt = (dt, at), where dt is the deliberation segment up to the decision behavior and at is the response. Two key properties: the decision is an externalized token segment that rules can read; the decision precedes the response, and the response is generated conditioned on it (binding), so training only the decision can change end-to-end behavior.

    Two rewards

    • External reward: judge J sees the full dialogue and the final response and determines whether it was jailbroken; R(cT) = 1 if and only if J judges it SAFE, where SAFE includes refusal or safe redirection; harmless dialogues count as SAFE only if helpfully complied with, to penalize over-refusal.
    • Self-verification reward: training dialogues grow from seeds with known properties, and the target decision g(b) is refuse for harmful sessions and comply for harmless ones; R(cT) = 1 if and only if the label parsed at the final turn equals g(b) and the response is non-empty. The non-empty condition prevents the degenerate optimum of "outputting only refuse without answering".
    • The self-verification setting excludes SAFE-REDIRECT: the authors say that in harmful sessions both refuse and safe-redirect are defensible commitments, and a rule that reads only the label cannot specify a unique target, so this setting is limited to a binary decision space.

    Gradient scope

    • Full gradient (default): all generated tokens in the final turn are trainable, and response style adjusts together with the decision.
    • Partial gradient: only tokens before the decision boundary are trainable, and response tokens are masked to 0; the authors say its feasibility depends on the binding property, and it largely preserves the base model's response style. The boundary is located by binary search mapping the character-level boundary to token indices.

    Training algorithm

    • Optimized with GRPO; per-turn length budget: each turn's response is capped at L tokens; the authors say that with only a trajectory-level budget, the policy inflates early easy-to-answer turns and crowds out later hard turns, and the per-turn cap closes this reward speculation path.
    • Outcome supervision: reward is given only at the final turn, R(τ) = r(yT ∣ x1:T, y1:T−1); the authors say multi-turn jailbreaks are delivered at the final turn, and process-level signals would be dominated by easy-to-answer turns.
    • Implementation (Appendix C): LoRA (rank 32, α=32), Adam learning rate 1e-4, KL coefficient 0.001, entropy bonus 0; per-turn generation cap 2048 tokens, at most 7 defense turns, n=16 rollouts sampled per prompt; Qwen3's native reasoning channel is enabled.

    Training data

    • Harmful pool: five multi-turn attack families instantiated on the same set of AdvBench behaviors (SEMA, Crescendo, FITD, Echoing, Hidden Intention Streamline), with behaviors split into disjoint training/test sets, and each training behavior randomly assigned one family.
    • Harmless pool: UltraChat-200k multi-turn dialogues, matched to harmful dialogues in surface form and depth; the two pools are mixed in equal proportion to suppress over-refusal.
  4. What experiments did the paper conduct?

    Controlled comparison across two backbones, 14 benchmarks, and 6 judges: the authors claim the two variants' average SAFE rate of 90.4%–93.4% exceeds all baselines, at the cost of increased over-refusal.

    Experimental Setup

    • Models tested: two backbones, Llama3.1-8B-Instruct and Qwen3-8B; each reward variant reports two conditions, unaligned (base weights + framework prompt) and aligned (after RL training).
    • Baselines: SFT, DPO, SDGO, MTSA, MUSE-D, X-Boundary, all trained on the same data as this paper; on Qwen3-8B, each of the four weight-trained baselines requires one configuration change to exceed the base (Appendix C), whereas this paper's recipe is unchanged across backbones.
    • Safety benchmarks: AdvBench (source of behaviors for training attacks), HarmBench (source of unseen behaviors), MultiBreak, CoSafe, MHJ (three out-of-training multi-turn benchmarks); sample sizes n=516, 785, 500, 1400, 537 (Table 3).
    • Utility and helpfulness: MMLU, HellaSwag, Winogrande, GSM8K, TruthfulQA, IFEval (completion format, bypassing the chat template, Figure 2); XSTest (n=250) and XSB (n=340) measure over-refusal, MT-Bench (160 per-turn scores) measures dialogue quality.
    • Judges: main judge gpt-5-mini, given the full conversation and target behavior, outputs a binary harmful judgment; Appendix E.3 reports five out-of-training judges (gpt-oss-20b, StrongREJECT, Llama-Guard-3, Qwen3Guard, HarmBench-13B).
    • Statistics: Table 3 gives 95% confidence intervals (Wilson for safety and over-refusal, behavior-clustered bootstrap for averages).

    Main Results

    Table 1 (judge gpt-5-mini, SAFE % ↑; Avg. is the mean over the five safety benchmarks):

    Swipe horizontally to read the table

    MethodLlama Avg.Llama MHJQwen Avg.Qwen MHJQwen XSB ↓
    Base59.938.046.122.315.6
    SFT75.955.159.935.640.9
    DPO69.446.780.253.639.4
    SDGO60.639.147.933.015.9
    MTSA69.247.184.870.953.5
    MUSE-D67.245.385.761.540.9
    X-Boundary86.352.064.627.449.1
    Self (aligned)90.458.593.475.256.8
    External (aligned)92.971.187.663.142.6
    • The authors claim this paper's method has the highest average safety rate on both backbones and leads in every safety column; on Llama the advantage comes mainly from the suite the strongest baseline handles worst (especially the human-constructed MHJ), while on Qwen it is a uniform lead across all five suites.
    • Cost: utility is essentially unchanged (all methods lie within about one joint interval of the base, Table 3), while over-refusal rises—the authors state that the aligned defenders' over-refusal increase relative to the base exceeds the confidence interval and is a real cost; on MT-Bench this paper's models remain competitive (e.g., Qwen self reward 7.36 vs. base 6.08).
    • Interval interpretation (Table 3): on Qwen, self reward's average lead over MUSE-D is about nine half-widths and the per-suite intervals are disjoint, but on MHJ its interval overlaps with MTSA's; external reward's average lead on Qwen is less than two half-widths, which the authors say should be read as comparable to the strongest baseline rather than ordered.

    Ablations (Figure 3, Llama backbone)

    • Removing the Decision row is the most damaging change: safety rates drop on every suite, and under human red-teaming even fall below the undefended base; removing Assessment costs much less, mainly manifesting as increased over-refusal. The authors conclude from this that what carries the defense is the commitment rather than the deliberation preceding it.
    • Per-turn reward loses safety but pushes over-refusal to the lowest; partial gradient trades a moderate rise in over-refusal for robustness, with answer quality unchanged.
    • Data mixture: automated safety saturates under all mixtures, and the mixture only determines where the policy sits on the trade-off frontier; oversampling harmful dialogues trades over-refusal and answer quality for red-team robustness, and the authors keep the balanced mixture.

    Intervention Probes (Appendix A, SEMA/AdvBench payload turns, n=104)

    • Forced-decision probe (Figure 6 left): forcing REFUSE makes the untrained unaligned models fully safe on both backbones; forcing COMPLY collapses the unaligned arms, while the aligned arms remain far above their unaligned counterparts—the authors state that training reinforces both the commitment and the reply after it.
    • Depth probe (Figure 6 right): after forcing REFUSE, k harmful tokens are prefilled; the unaligned arms lose most of their safety with a small amount of prefilling, and training yields post-decision depth on Llama but only marginal depth on Qwen3; all curves eventually decay, which the authors call an expected feature of a defense where "shallowness is by design," and attackers able to write arbitrarily to the assistant channel are outside the threat model.
    • Decoding sampling probe: 1,248 samples per arm; among unaligned Llama samples that flip to COMPLY, about two-thirds are judged jailbroken, while the aligned arms almost never sample COMPLY (flip rate at most 0.2%); on Qwen3 the three-label arms' 20–30% flips all occur between REFUSE and SAFE-REDIRECT, with attack success rate remaining zero.

    Other Analyses

    • KL analysis (Figure 4): the authors state that DKL(aligned ∥ unaligned) is not concentrated in the first few tokens but distributed over the assessment segment before the decision, peaking near the decision and then gradually vanishing; both aligned and unaligned diverge from the base at the first position.
    • Judge agreement (Figure 5, 3,738 conversations, seven judgment sources): all judge pairs are positively but only moderately correlated; the model's own Decision label correlates with each external judge to a degree comparable to inter-judge agreement, even though external judges cannot see the deliberation segment.
    • Multi-judge safety (Table 4): self reward is strictly highest in 22 of the 25 judge×benchmark cells on Qwen3-8B, the only cell where a baseline exceeds it being MHJ under Qwen3Guard (MTSA 81.8 vs. 78.8); external reward falls out of the top two in 9 cells. The authors state that the judge-independence conclusion applies only to self reward; averaged over five judges, self reward leads the strongest baseline by +7.1 (MultiBreak), +2.6 (MHJ), +3.2 (CoSafe).
  5. What are some points that can be further explored?

    The authors acknowledge that adaptive adversaries are not covered, that the ranking of the two rewards reverses with the backbone and the reason is unclear, and that judge agreement is only moderate.

    Limitations and Future Directions Noted by the Authors

    • Adaptive adversaries: the threat model is statically replayed scripted multi-turn attacks, and the authors explicitly state "adaptive adversaries acknowledged as an open limitation" (§5 Related Work).
    • Unpredictable ordering of reward variants: which of the self and external rewards is safer and which is more useful reverses with the backbone (Appendix B); the authors state that the experiments did not isolate the cause—the rewards differ both in what content they read and in whether a judge is in the loop, and the two backbones also differ in whether they have a native reasoning channel before the decision—and mark this as an open question.
    • A single judge cannot serve as ground truth: judges are only moderately correlated and disagree on a substantial fraction of samples (§4.2), and the authors use this to caution against treating any single judge's attack success rate as truth, and accordingly report multi-judge results in Appendix E.3.
    • Self-verifying reward is limited to binary decisions: a rule that reads only the label cannot specify a unique target between refuse and safe-redirect, and the three-label space is left to the judge reward (§2.3).
    • Shallow defense is by design: the depth probe shows that the protection does not withstand attackers able to write arbitrarily to the assistant channel (§A.2), and the authors state that this probe quantifies the boundary of the threat model.

    Scope of Experimental Coverage

    • The models tested are the two backbones Llama3.1-8B-Instruct and Qwen3-8B; training uses LoRA, and the recipe is unchanged across backbones (Appendix C).
    • Safety evaluation covers five benchmarks totaling 3,738 conversations (Table 3), with the main judge gpt-5-mini and five additional out-of-training judges (Table 4); the paper does not report agreement between judges and human judgments.
    • Intervention probes are run on 104 payload turns from SEMA/AdvBench (Appendix A); the decoding sampling probe uses 1,248 samples per arm.
    • Training attacks cover five multi-turn attack families, all instantiated on the same set of AdvBench behaviors; the harmless data is UltraChat-200k.
    • Utility evaluation is conducted in completion format, bypassing the chat template (Figure 2), which the authors state bypasses scaffolding and the format-concentrated component of alignment training.
  6. Summarize the main content of the paper

    Turns shallow alignment from a weakness into a defense mechanism: explicit commitment plus final-turn RL; the authors say multi-turn safety exceeds same-data baselines, at the cost of over-refusal.

    A paper on multi-turn jailbreak defense whose core stance is to turn the property of "shallow alignment" exploited by attackers into a defense mechanism.

    • Problem: single-turn refusal is already fairly reliable, but harmful requests spread across multiple turns still often succeed; the authors argue that existing training-based defenses rely on judges supervising the surface form of responses, and the model never learns an explicit, rule-checkable safety decision.
    • Method: Assess → Decide → Respond—each turn, first assess intent, commit to a parseable decision label, and then respond accordingly; GRPO gives reward only at the payload turn. The self-verification reward uses rules to compare the committed label against known properties of the session, requiring no judge or annotation; the external-reward version supports three labels (including SAFE-REDIRECT).
    • Main results (Table 1, gpt-5-mini judge): on Llama3.1-8B, the self/external variants achieve average SAFE rates of 90.4%/92.9%, and on Qwen3-8B, 93.4%/87.6%; the strongest baselines are X-Boundary at 86.3% (Llama) and MUSE-D at 85.7% (Qwen); on the hardest MHJ suite, Qwen self reward reaches 75.2%, versus 22.3% for the base model.
    • Costs and boundaries: utility is essentially unchanged, over-refusal rises (XSB for Qwen self reward rises from 15.6% to 56.8%); the intervals in Table 3 show that on some individual suites it is indistinguishable from the strongest baseline.
    • Mechanistic evidence: forcing REFUSE makes untrained models fully safe, and under forced COMPLY aligned models remain highly safe (Appendix A); KL analysis shows the training effect is distributed in the assessment-decision segment rather than the opening refusal prefix (Figure 4); the ablation removing the Decision line causes the greatest damage.
    • Authors' conclusion: the structural property of alignment can be engineered into a defense, and commitment provides a scalable alternative to judge supervision; adaptive adversaries and the question of ranking the two rewards are left as open problems.

Related papers

Show 3 more papers
Read originalarxiv.org