TRACE: Trajectory Return Attribution and Contrastive Erasure for multi-turn safety
TRACE: Trajectory Return Attribution and Contrastive Erasure for Multi-Turn Safety
The authors propose TRACE, which uses single-turn preference data for token-level alignment training; on 5 models × 7 multi-turn attacks, 35 pairs in total, ASR is lowest in all of them, and utility drops by no more than 1.23 points.
Defender: fine-tune open-source models with 400 single-turn preference triples (harmful request x, preferred safe response y+, refusal response y−), without using multi-turn data. The attacker conducts multi-turn jailbreaks at deployment time: MHJ and SafeDialBench replay fixed user turns, while ActorAttack, Crescendo, STAR, RACE, and X-Teaming use WizardLM-2-8x22B to adaptively generate based on the target's responses; the attack succeeds if any turn is judged successful.
- Safety-aligned LLMs can often refuse single-turn harmful requests, but once the same goal is spread across multiple turns, they comply. Preference objectives only score the complete response under a fixed context; the authors call this safety failure across interaction histories trajectory fragility.
- TRACE performs token-level training on single-turn preference triples: the preference branch weights each token using a discounted return constructed from the log-probability difference between a frozen reference model and its refusal-ablated copy; the refusal branch, at positions where the reference-normalized log-probability difference is largest, mixes the observed token with high-probability policy alternatives into an erasure target; then a gradient-norm penalty replaces the retain set.
- On 5 open-source models (1B–27B) × 7 multi-turn attacks, 35 pairs in total, TRACE has the lowest ASR in all of them; average multi-turn ASR drops from 52.81% for the compared methods to 28.98%, and MMLU and HellaSwag drop by at most 1.23 points. On Llama-3.1-8B, the XSTest safe-prompt refusal rate rises from 5.48% for the original model to 17.23% (ξ=0.05).
- The theoretical analysis relies on assumptions such as coverage and transfer slack; the paper does not estimate ρ, ε, κ, and the authors state that this bound does not constitute a proof of reduction relative to the base policy's own trajectory risk. The over-refusal measurement covers only Llama-3.1-8B; the GPT-4o-mini judge has annotation inconsistencies, and the paper does not give the judge error rate.
- Evaluated models
- Gemma-3-1BLlama-3.1-8BQwen-3.5-9BQwen-3.5-27BQwen-3.6-27B
- Benchmarks
- SORRY-BenchHEx-PHIMHJSafeDialBenchActorAttackCrescendoSTARRACEX-TeamingGCGAutoDANMMLUHellaSwagXSTest
- Metrics
- ASRMMLU accuracyHellaSwag accuracyXSTest refusal rate
The researchers propose a token-level training objective for multi-turn safety, TRACE (Trajectory Return Attribution and Contrastive Erasure), to reduce the attack success rate of large language models under multi-turn attacks. Addressing the problem that safety-aligned models refuse in a single turn but comply when the same harmful goal is spread across multiple turns, the paper gives a sufficient condition characterizing when suppression in the supervised single-turn context can constrain multi-turn trajectory risk. The method weights each token on safe responses by the discounted return of the refusal-attributable advantage, obtained by comparing a frozen reference model with its refusal-ablated copy, so that earlier response tokens can receive credit from later refusal-related evidence; at high-gap positions of refused responses, observed tokens are combined with policy-chosen alternatives into an erasure objective, and a gradient-norm penalty replaces the retention set.
Paper analysis
What problem does this paper attempt to solve?
What the authors aim to solve: when training only with single-turn preference data, how to constrain the safety risk that accumulates over multi-turn dialogue trajectories.
Core problem: safety alignment trained only with single-turn preference supervision cannot constrain the risk that accumulates over multi-turn dialogue trajectories.
- Problem setting: the authors state that safety-aligned LLMs often refuse single-turn harmful requests, but comply when the same goal is spread across multiple turns, or even when the model's own responses are reused by later prompts (Figure 1 gives an illustration). The authors call this safety failure across interaction histories trajectory fragility.
- Shortcomings of existing methods (as stated in the paper): response-level preference objectives such as RLHF and DPO only score the complete response under a fixed context, and their training loss itself cannot control risk on unseen histories; multi-turn training only covers the policies appearing in its dialogues; retain-set losses only protect the behaviors covered by their benign samples.
- Core observation: different attack histories may lead to similar harmful responses, so single-turn requests and multi-turn histories can share harmful intent; Proposition 3.1 of the paper states that, without a coverage assumption, the static preference loss has no non-trivial upper bound on trajectory risk.
- What the paper proposes: a conditional trajectory risk bound (Theorem 3.3, connecting local suppression at single-turn contexts to multi-turn risk via coverage, transfer slack, and leakage), and a method TRACE (TAW + RLPE + gradient-norm penalty) that turns it into a token-level objective.
- Threat model:
- Defender's perspective: fine-tune open-source models with 400 single-turn preference triples (harmful request x, preferred safe response y+, refusal response y−), without using multi-turn data.
- Attacker's capability: launch multi-turn jailbreaks at deployment—MHJ and SafeDialBench replay fixed user turns; ActorAttack, Crescendo, STAR, RACE, X-Teaming adaptively generate based on target responses with WizardLM-2-8x22B.
- Victim system: 5 open-source instruction models (Gemma-3-1B to Qwen-3.6-27B), greedy decoding.
- Success criterion: if any turn is judged successful by the judge, the entire attack succeeds.
What related research exists?
Related work falls into four groups: multi-turn jailbreaks and trajectory safety, preference alignment and targeted suppression, utility preservation and alignment tax, and the baselines of this paper.
Multi-turn jailbreaks and trajectory-level safety
- Crescendo (Russinovich et al., 2025) escalates an initially benign request turn by turn; ActorAttack (Ren et al., 2024) and MHJ (Li et al., 2024) decompose a harmful goal into harmless sub-questions; Siege, RACE, X-Teaming (Zhou & Arel, 2025; Ying et al., 2025; Rahman et al., 2025) adaptively search or plan dialogue branches; STAR (Li et al., 2026) studies state-dependent failures across turns.
- SafeDialBench (Cao et al., 2025) and MultiBreak (Song et al., 2026) are cited as confirming that per-turn safety does not compose into trajectory-level safety; Yang et al. (2025) state that most of the advantage of multi-turn attacks can be recovered by variants of single-turn attacks.
- Qi et al. (2025a) attribute part of the problem to shallow alignment concentrated in the first few tokens of the response; multi-turn safety training uses dialogue data such as SafeMTData (Ren et al., 2024; X-Boundary's multi-turn defense set is also built from it, Lu et al., 2025).
Preference alignment and targeted suppression
- RLHF (Ouyang et al., 2022; Bai et al., 2022) and DPO and its variants (Azar et al., 2024; Ethayarajh et al., 2024; Hong et al., 2024; Meng et al., 2024) score the entire response; SafeDPO (Kim et al., 2025), B-DPO (Zhao et al., 2026a), and Uni-DPO (Peng et al., 2026) keep this granularity, while X-Boundary (Lu et al., 2025) acts on internal representations.
- DOOR (Zhao et al., 2025) combines refusal training after harmful prefixes, NPO unlearning on harmful responses (Zhang et al., 2024), and a retain loss; W-DOOR adds token weights e^{rt/τ}, and its reported implementation computes rt with a DPO-trained scorer. TIS-DPO (Liu et al., 2025) estimates token importance from the truncated log-ratio of a contrast model.
- The paper states the difference from this work: TRACE's token weights use the discounted return of the reference–ablation log-ratio rather than a function of each token's own ratio (Proposition 4.1 states that this allows orderings that non-decreasing functions cannot achieve); the contrast model is a refusal-ablated copy (Arditi et al., 2024), which the paper calls the training-free counterpart of the W-DOOR variant tested by Zhao et al. (2025); the logistic transform on the refusal side is related to SimNPO (Fan et al., 2025).
Utility preservation and alignment tax
- Safety fine-tuning often loses general capability and causes over-refusal (Huang et al., 2025; Niu et al., 2025; Zhao et al., 2026b); over-refusal is measured with XSTest (Röttger et al., 2024).
- Common practice is a preservation loss on a retain set (NPO+RT, DOOR) or KL anchoring to the reference policy; the paper states that both only protect the behaviors covered by their data, and their effect depends on data selection (Chang & Lee, 2025; Slocum et al., 2025). TRACE instead uses a gradient-norm penalty on the alignment batch itself, following the penalty of Zhao et al. (2022) (related to SAM, Foret et al., 2021).
Baselines and benchmarks: the baselines are SFT, DPO, B-DPO, Uni-DPO, X-Boundary, DOOR, W-DOOR, all using the same 400 training triples; evaluation uses 7 multi-turn protocols (MHJ, SafeDialBench, ActorAttack, Crescendo, STAR, RACE, X-Teaming), single-turn attacks GCG and AutoDAN, and utility uses MMLU and HellaSwag.
The authors position this paper as: learning from single-turn request–response preference pairs, using Theorem 3.3 to give conditions under which local suppression connects to a trajectory risk budget, and evaluating on multi-turn attacks.
How does the paper address this problem?
TRACE refines the supervision of single-turn preference triples to the token level: the preference branch is weighted by discounted return, the refusal branch performs local proxy erasure, and a gradient-norm penalty is added.
Overall approach: on single-turn preference data, split response-level supervision into token-level—the preference branch aggregates subsequent evidence to assign weights to earlier tokens, the refusal branch suppresses only at high-risk positions, and a gradient-norm penalty replaces the retain set. The method is named TRACE (Trajectory Return Attribution and Contrastive Erasure), with the overall objective LTRACE(θ) = LAlign(θ) + λret |∇θ LAlign(θ)|, i.e., the alignment loss plus a gradient-norm penalty; LAlign = LTAW + LRLPE.
Theoretical Motivation: Trajectory Risk and the Single-Turn Transfer Principle
- Trajectory risk is defined as the sum of the expected number of unsafe responses produced at each turn (Eq. 1). Proposition 3.1 states: without a coverage assumption, the static preference loss has no nontrivial upper bound on trajectory risk; two policies with the same preference loss can differ in trajectory risk by T; a favorable margin does not guarantee a low unsafe probability at every safety-critical token context.
- Assumption 3.2 (Intent-Sufficient Local Risk) requires that the difference in unsafe mass between a covered context and its supervised representative be bounded by a slack ε. Theorem 3.3 gives RT(πnew) ≤ [1 − ρ(1 − e2ε − Δ)] Bbase + L, meaning: when leakage L=0, the risk budget shrinks if and only if the coverage share ρ>0 and the suppression level ∆>2ε; where B_base is the base policy's unsafe mass evaluated on the contexts visited by the new policy.
- The authors explicitly state: this shrinkage is relative to this risk budget and does not constitute a proof of reduction relative to the base policy's own trajectory risk; the paper does not report estimates of ρ, ε, κ.
TAW: Trajectory-Aware Adaptive Weighting (Preference Branch)
- A frozen reference model πref and its refusal-ablated copy πabl (publicly available, training-free, method of Arditi et al., 2024) are used to compute each token's step advantage rj = log πref(yj|cj) − log πabl(yj|cj).
- rj is aggregated from the current position into a discounted return Gj = rj + λGj+1 (λ∈[0,1)), so that refusal-related evidence in later parts of the response is counted in the weights of earlier tokens; Proposition 4.1 states that any non-decreasing transformation of the immediate advantage cannot rank earlier tokens higher when later advantages are larger, whereas the return can.
- Within each response, the return is standardized by median and MAD, truncated, and compressed via tanh into wjtraj, then multiplied by wjcap, which downweights predictable tokens, and by two response-level factors, truncated at the 5/95 percentiles of the minibatch and normalized within the response to unit mean; the paper gives an explicit upper bound on the token weight ratio under parameter constraints (Proposition C.1).
RLPE: Risk-Localized Proxy Erasure (Refusal Branch)
- Up to kproxy positions with the largest reference-normalized log-probability difference δj are selected; the authors state that this signal is a contrastive selection signal rather than a semantic harmfulness label.
- At each selected position, the observed refusal token is mixed with the kproxy alternative tokens with the highest probability under the current policy into a target distribution sj (mixing coefficient λproxy), on the grounds that suppressing only the observed token would leave high-probability harmful alternatives.
- The cross-entropy at the selected positions is averaged and then subjected to a reference-independent logistic transformation (related to SimNPO); the authors note that this aggregate objective does not guarantee that every proxy probability decreases, and Theorem 4.4 plugs the proxy coverage κ into the trajectory risk bound under an additional per-token suppression assumption (Eq. 18).
Gradient-Norm Penalty and Training Procedure
- No retain data is used; the alignment gradient norm is penalized directly; Proposition 4.2 states that the gradient norm controls single-step parameter drift, thereby bounding the change in benign outputs. Following Zhao et al. (2022), one extra forward gradient replaces the Hessian-vector product: the gradient is computed again at θ+ = θ + ξ·ĝ, with mixing weight α = λret/ξ (Lemma 4.3).
- Key parameters (Table 7): λ=0.5, kproxy=10, λproxy=0.5, ξ 0.05 for full fine-tuning / 0.2 for LoRA, α=0.9, learning rate 1e-5; 1B–9B full fine-tuning for 2,000 steps (10 epochs), 27B model trained with rank-16 LoRA for 2 epochs. Algorithm 1 (Appendix D.5) gives the complete procedure.
What experiments did the paper conduct?
5 models × 7 multi-turn attacks: TRACE has the lowest ASR on all 35 pairs, with an average multi-turn ASR of 28.98% and a utility drop of at most 1.23 points.
Experimental setup
- Models evaluated: Gemma-3-1B, Llama-3.1-8B, Qwen-3.5-9B (full fine-tuning), Qwen-3.5-27B, Qwen-3.6-27B (rank-16 LoRA, 4-bit base).
- Training data: 400 single-turn preference triples, with harmful requests from SORRY-Bench and HEx-PHI, no multi-turn data; all methods share the same set of triples.
- Attacks and scale: 7 multi-turn protocols totaling 4,083 nominal entries—ActorAttack 600, Crescendo 150, MHJ 537, SafeDialBench 2,037, STAR 150, RACE 450, X-Teaming 159 (Table 8); MHJ and SafeDialBench replay fixed turns, and the other five are adaptively generated with WizardLM-2-8x22B. Single-turn attacks are GCG and AutoDAN (transfer attacks, using the HarmBench prompts released by DOOR).
- Judge: GPT-4o-mini (2024-07-18, temperature 0) judges on the full dialogue context and cannot see the training method; MHJ uses the HarmBench Llama-2-13B classifier. An attack succeeds if any turn is judged successful.
- Metrics: ASR (%), MMLU and HellaSwag accuracy (%), XSTest safe-prompt refusal rate.
- Repeats: all reported results are the mean of 5 experimental runs, training seed 42.
Main results
According to Table 1 (three smaller models) and Table 2 (Qwen-3.5-27B), multi-turn ASR (%) and utility:
Swipe horizontally to read the table
Model Method 7-attack mean ASR Lowest baseline mean MMLU HellaSwag Gemma-3-1B TRACE 26.55 53.13 (W-DOOR) 40.06 58.88 Llama-3.1-8B TRACE 27.14 58.91 (DPO) 66.71 78.27 Qwen-3.5-9B TRACE 27.31 49.90 (W-DOOR) 78.44 76.82 Qwen-3.5-27B TRACE 29.49 42.76 (W-DOOR variant) 84.58 83.40 - The authors report: TRACE has the lowest ASR on all 35 model×attack pairs (25 adaptive + 10 replay); the 5-model average multi-turn ASR is 28.98%, versus 52.81% for the compared methods (51.88% when selecting the lowest baseline per model).
- Relative to the best baseline in each column, the largest single gap is on Llama-3.1-8B X-Teaming (86.79%→33.96%), and the smallest is on Qwen-3.5-9B SafeDialBench (21.15%→14.58%).
- Exceptions: X-Teaming remains difficult, with a TRACE five-model mean of 53.96%, reaching 76.73% on Qwen-3.5-9B; on Qwen-3.6-27B every baseline raises the mean above the original model (47.14%) (54.68%–59.40%), while TRACE lowers it to 34.43% (Table 9).
- Utility: MMLU and HellaSwag drop by at most 1.23 points relative to the original model, and both rise on Gemma-3-1B; on Llama-3.1-8B, TRACE's MMLU loss (1.10 points) is larger than that of DOOR and W-DOOR with a retain set.
Ablation (Table 3, Llama-3.1-8B, 7-attack mean)
- Without TAW: the mean rises from 27.14% to 50.37%; adding TAW to DPO lowers it from 58.91% to 41.25%.
- Without RLPE: the mean rises to 34.91%, and GCG rises from 3.27% to 7.04%.
- Without the gradient-norm penalty (ξ=0): the mean drops to 20.89% (lower), but MMLU/HellaSwag drop by 1.29/2.91 points relative to full TRACE; replacing it with a retain loss on 400 Alpaca examples raises the mean to 36.50%.
- Replacing πabl with πref: all advantages and returns become zero, the trajectory weight degenerates to 1, and the mean rises to about 44% (see Table 3 for each attack column).
Over-refusal and judge consistency
- XSTest (Table 13, Llama-3.1-8B): the original model's refusal rate is 5.48%, baselines range from 5.41% (DOOR) to 12.84% (X-Boundary); TRACE with the default ξ=0.05 is 17.23%, and reaches 47.52% at ξ=0.2. The authors state that this presents a trade-off between lower multi-turn ASR and more refusals of safe prompts.
- Judge: the authors report annotation inconsistencies in GPT-4o-mini—in the Tom letter example in Appendix G.6, the target model refuses and the judge's explanation also describes a refusal, but the label is judged as a successful attack; the paper states that these examples cannot be used to estimate the judge error rate. When MHJ uses the HarmBench classifier, TRACE improves by 21.93–39.35 percentage points relative to the best baseline for each model, which the authors state is evidence of improvement under a second judging protocol.
Other ablations and analyses
- Discount λ (Table 10, Llama-3.1-8B): mean ASR is 43.55% at λ=0, 27.14% at λ=0.5, and 40.26% at λ=0.9.
- Budget kproxy (Table 11): 38.65% at k=0 (NPO special case), 27.14% at k=10, rising to 46.60% at k=20; the authors state that this sweep simultaneously changes position selection, proxy support, and loss normalization.
- Radius ξ (Table 12): at ξ=0.01 the mean is 31.32% and utility is within 0.2 points of the original model; at ξ=0.2 the mean is 23.63% and MMLU is 66.16%.
- Training time (Table 3): full TRACE 2.98 hours, the retain-loss variant 2.42 hours, no penalty 2.09 hours (single H200).
What are some points that can be further explored?
The authors state that the theoretical bounds rely on unestimated assumptions and that reviewer consistency is lacking; the experiments cover 5 open-source models and 7 multi-turn protocols.
Limitations and future directions noted by the authors
- The theoretical contraction is relative to the base policy risk budget evaluated on new policy contexts; the authors state explicitly that it does not constitute a proof of reduction relative to the base policy's own trajectory risk (Sec. 3, Appendix C.1.4).
- Quantities such as the coverage share ρ, the transfer slack ε, and the proxy coverage κ are not estimated; Appendix C.1.2 states that the analysis assumes a fixed unsafe token set and satisfies emission, and does not report estimates of these quantities.
- The experiments measure dialogue-level ASR, whereas the per-token suppression condition of Theorem 4.4 (Eq. 18) is an additional premise of the bound; Appendix E states that ASR measurement cannot identify coverage or per-token suppression.
- Automatic judging may have false positives: Appendices F.3 and G.6 give examples of target refusals judged as successful attacks, and state that these examples cannot be used to estimate the judge error rate.
- The over-refusal measurement covers only one model (Llama-3.1-8B); Appendix G.5 states that the two multiple-choice utility scores do not measure this behavior.
- The kproxy sweep simultaneously changes position selection, proxy support, and loss normalization; Appendix G.3 states that it measures the joint effect, and that the k=0 row additionally switches to the NPO special case, so it is not an isolated proxy smoothing ablation.
Experimental coverage
- The models tested are 5 open-source models: Gemma-3-1B, Llama-3.1-8B, Qwen-3.5-9B, Qwen-3.5-27B, Qwen-3.6-27B; hyperparameters were selected on Llama-3.1-8B, and the other models use similar configurations (Appendix F.2).
- The training data are fixed at 400 single-turn preference triples (SORRY-Bench + HEx-PHI); no multi-turn training data were constructed or used (Appendix F.1).
- The attack evaluation consists of 7 multi-turn protocols (nominally 4,083 items) plus two single-turn transfer attacks, GCG and AutoDAN; the generative model for the adaptive attack is WizardLM-2-8x22B, and the target model uses greedy decoding (Appendix F.3).
- The judge is GPT-4o-mini (MHJ uses the HarmBench classifier); the paper does not report agreement between the judge and human judgment.
- Results are the mean of 5 runs, with the training seed fixed at 42; the paper does not report variance across runs or statistical tests.
Summarize the main content of the paper
TRACE uses single-turn preference data for token-level alignment; on 5 models × 7 attacks its ASR is lowest in all of them, and utility loss does not exceed 1.23 points.
TRACE is a multi-turn safety defense paper: the authors state that aligned models can refuse single-turn harmful requests but fail in multi-turn dialogue (trajectory fragility), and that response-level preference objectives themselves cannot constrain trajectory risk.
- Problem: when training only with single-turn preference triples, how to reduce risk under multi-turn attacks. Theoretically, Proposition 3.1 states that the static preference loss has no non-trivial upper bound on trajectory risk; Theorem 3.3 gives a conditional bound that connects local suppression to a trajectory risk budget via coverage, transfer slack, and leakage.
- Method: the preference branch constructs a discounted return from the log-ratio of a frozen reference model and its refusal-ablated copy, and weights tokens (TAW); the refusal branch, at positions where the reference-normalized gap is largest, mixes the observed token with high-probability policy alternatives into an erasure target (RLPE); then a gradient-norm penalty on the alignment batch itself replaces the retain set.
- Main results: on 5 open-source models (1B–27B) × 7 multi-turn attacks, 35 pairs in total, TRACE has the lowest ASR in all of them; average multi-turn ASR drops from 52.81% for the compared methods to 28.98%, and MMLU and HellaSwag drop by at most 1.23 points (Table 1, 2, 9).
- Ablation: on Llama-3.1-8B, removing TAW raises mean ASR from 27.14% to 50.37%, and removing RLPE raises it to 34.91%; removing the gradient-norm penalty lowers the mean to 20.89% but utility drops more (Table 3).
- Costs and exceptions: the XSTest refusal rate rises from 5.48% for the original model to 17.23% (ξ=0.05), and reaches 47.52% at ξ=0.2; on X-Teaming, TRACE's five-model mean is still 53.96%.
- Authors' conclusion: under the reported configurations, single-turn supervision can transfer to the evaluated multi-turn histories; at the same time, the authors explicitly state that the theoretical bound does not constitute a proof of reduction relative to the base policy's own trajectory risk, and that ρ, ε, κ are not estimated.