Skip to content

TraceRouter++: Causal Semantic Routing Realignment Based on Optimal Transport

TraceRouter++: Causal Semantic Routing Realignment via Optimal Transport

arXiv: Interpretability· arXiv:2611.04878· IEEE Transactions on Pattern Analysis and Machine Intelligence 2026· Chuancheng Shi, Shangze Li, Wenhua Wu, Hengzhi Chen, Fei Shen, Li Zhuo, Zhiyong Wang, Tat-Seng ChuaThe University of Sydney; Nanjing University of Science and Technology; National University of Singapore; Beijing University of TechnologyIndexed by this site · 16 days ago· Originally publishedDefenses

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

The authors propose a routing-level safety intervention framework, TraceRouter++, and claim that on LLMs, MLLMs, and DMs, DSR reaches up to 99.5±0.3%, with utility loss not exceeding 0.6 percentage points.

Threat modelThe defender intervenes on LFM internal activations at inference time to resist jailbreaks and harmful content generation: LLMs face gradient driven attacks such as GCG and AutoDAN and pattern injections such as Template and Prefill; MLLMs face FigStep on the image side and JailbreakV-28K on the text side; DMs face P4D and Ring-A-Bell adversarial prompts. Attackers can construct arbitrary input prompts, and the paper does not restrict the attacker's knowledge of model internals.

Questions
Existing safety interventions are mostly based on a locality assumption, hard-suppressing isolated neurons or features; the authors argue that harmful semantics propagate along cross-layer routing, and that local ablation is easily bypassed by complex jailbreaks and damages general capabilities.
Methods
TraceRouter++ operates as Discover–Trace–Realign: it uses attention divergence to locate the sensitive starting layer, decomposes routing features with a Top-K SAE, traces cross-layer causal paths with FIS, and then uses coordinate-wise marginal optimal transport to softly realign the unsafe activation distribution to a safe reference distribution.
Experiments and results
Evaluated on LLMs, MLLMs, and DMs: LLaMA3-8B-Instruct achieves an average DSR of 99.5±0.3%, with utility degradation not exceeding 0.6 percentage points; however, under P4D attack DSR is 76.0±1.3%, and under defense-aware adaptive GCG DSR drops to 73.4±2.1% (LLaMA3) and 52.4±1.1% (Mistral).
Limitations and future work
In Appendix B, the authors note that the method does not have general robustness against defense-aware adaptive GCG, and that cross-model adaptive robustness is a future direction; Appendix A reports failure cases where Prefill-type attacks can override fixed-strength interventions, and the influence of the semantic composition of the safety reference set was not independently studied.
Experimental setupLLaMA3-8B-Instruct、Mistral-7B-Instruct et al. · AdvBench、HarmBench et al. · DSR、ORR et al.
Evaluated models
LLaMA3-8B-InstructMistral-7B-InstructLLaVA-1.5-7BMiniGPT-4-7BDeepSeek-VL2Stable Diffusion 1.4FLUX.1-DevAltDiffusionShow-o2
Benchmarks
AdvBenchHarmBenchGCGAutoDANTemplatePrefillXSTESTGlobal-MMLU-LiteFigStepJailbreakV-28KMMBenchI2P-NI2P-VP4DRing-A-BellMS COCO
Metrics
DSRORRAccuracyCLIP ScoreFID
AI summaryResearchers propose TraceRouter++, a routing-level safety intervention framework that traces the causal semantic routing pathways activated by harmful inputs and realigns their unsafe activation states. The framework uses attention divergence to locate sensitive onset layers, maps to downstream causal circuits via sparse autoencoders (SAEs) and feature influence scores (FIS), and uses optimal-transport soft realignment to steer unsafe activations toward safe references. Experiments on LLMs, multimodal large models, and diffusion models show that TraceRouter++ outperforms the evaluated baselines, improving adversarial robustness while maintaining a favorable safety-utility balance.

Researchers propose TraceRouter++, a routing-level safety intervention framework that traces the causal semantic routing pathways activated by harmful inputs and realigns their unsafe activation states. The framework uses attention divergence to locate sensitive onset layers, maps to downstream causal circuits via sparse autoencoders (SAEs) and feature influence scores (FIS), and uses optimal-transport soft realignment to steer unsafe activations toward safe references. Experiments on LLMs, multimodal large models, and diffusion models show that TraceRouter++ outperforms the evaluated baselines, improving adversarial robustness while maintaining a favorable safety-utility balance.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors argue that local neuron suppression is easily bypassed by jailbreaks and harms utility, and propose a routing-level soft realignment defense.

    Core question: how to intercept harmful semantics propagating along cross-layer routes in large foundation models (LFMs) at inference time without harming general capabilities.

    • Problem setting: The authors state that LLMs, MLLMs, and DMs all face the risk of adversarial manipulation; existing safety interventions mostly rely on the locality hypothesis, hard-suppressing isolated neurons or features.
    • Shortcomings of existing methods: The authors argue that harmful semantics exist as distributed, cross-layer circuits; local ablation is like an isolated roadblock, and under complex jailbreaks harmful information can recover via alternative paths; forcibly cutting entangled components also disrupts benign computation and reduces general utility (Figure 1).
    • Core hypothesis: The paper formally proposes the routing-level representation hypothesis—safety-critical semantics are encoded and transmitted via distributed causal routing circuits—and therefore models safety intervention as a distribution alignment problem on a routing subspace, using optimal transport (OT) for soft realignment.
    • Proposed method: TraceRouter++ is proposed, which follows the three stages Discover–Trace–Realign to locate the sensitive starting layer, trace causal routing paths, and perform OT-driven soft routing realignment; it is an extended version of the prior work TraceRouter [67].
    • Threat model:
      • Attacker goal: induce the model to output harmful content (LLM/MLLM jailbreaks, DM generation of nude or violent images).
      • Attack methods: on LLMs, gradient driven attacks such as GCG and AutoDAN and pattern injections such as Template and Prefill; on MLLMs, FigStep on the image side and JailbreakV-28K on the text side; on DMs, P4D and Ring-A-Bell adversarial prompts.
      • Attacker knowledge: the paper does not explicitly restrict it; Appendix B additionally tests defense-aware adaptive GCG that scores against the intervened model.
      • Victim systems: LLaMA3-8B-Instruct, Mistral-7B-Instruct, LLaVA-1.5-7B, MiniGPT-4-7B, DeepSeek-VL2, Stable Diffusion 1.4, etc., with TraceRouter++ inference-time intervention loaded.
  2. What related research exists?

    Related work falls into three groups: LFM safety intervention, causal routing analysis, and optimal transport in representation learning.

    LFM safety interventions

    • Training-time alignment RLHF [52], DPO [57] and inference-time defenses circuit breaker (CB) [94], safety-specific neuron ablation [90]: the former optimize generation preferences, the latter explicitly disrupt malicious representations.
    • DM interventions: inference-time guidance SLD [62], closed-form parameter editing ESD [19], UCE [20], MACE [46] for concept erasure; MLLM defenses ECSO [26] (image-to-text), CoCA [21] (constitutional calibration), SafePTR [10] and SGM [74] (token/neuron-level interception).
    • The paper points out that these lightweight editing and fine-grained methods mostly implicitly rely on the locality assumption, and the authors say such local erasure is often bypassed by complex adversarial prompts [54] and introduces side effects [61]; this criticism targets this phenomenon, and the paper does not name each work individually.

    Causal routing analysis of neural representations

    • Mechanistic interpretability research holds that semantic behavior arises from cross-layer routing paths rather than isolated nodes: influential neuron paths [75] characterize how concepts are expressed via structured routing in a superpositional form; DM studies [30], [9] show that textual concepts guide image synthesis via multi-layer attention trajectories.
    • Representation steering and concept erasure methods [93], [7] control behavior by modifying activation directions or projecting concept information. The paper says most safety interventions have not yet integrated causal routing paths into defense mechanisms, and TraceRouter++ combines cross-layer routing identification with per-coordinate OT.

    Optimal transport in representation learning

    • OT [53] and the Wasserstein distance [5] have been used for domain adaptation [13], generative modeling [5], and cross-modal alignment [2]; recently OpT-DeUS [8] fuses Transformer blocks, OTPO [40] uses unbalanced OT for token weighting, and MOT [64] infers soft coupling between layers.
    • The paper says the use of OT for internal causal routing realignment remains underexplored, and existing safety interventions are mostly rigid ablation or orthogonal projection [93], [7].

    Baselines and benchmarks: LLM baselines are DeepAug [55], CB [94], DeRTa [87], HumorReject [81], LEACE [7], TraceRouter [67], with benchmarks AdvBench/HarmBench plus GCG, AutoDAN, Template, Prefill attacks; MLLM baselines are FigStep [25], CoCA, ECSO, AdaShield [76], Immune [23], SafePTR, with benchmarks JailbreakV-28K and FigStep; DM baselines are ESD, UCE, CA [37], SLD-Med, MACE, RECE [24], SPM [48], DuMo [28], SNCE [29], with benchmarks I2P-N/V, P4D, Ring-A-Bell. The authors position this paper as a general framework combining routing-level mechanistic understanding with OT soft realignment.

  3. How does the paper address this problem?

    Discover–Trace–Realign: locate the starting layer, decompose features with SAE, trace paths with FIS, and perform OT soft realignment.

    Overall approach: safety intervention is formalized as a constrained distribution alignment problem on a causal routing subspace, mapping the marginal distribution of unsafe activations along identified malicious routing paths toward a safe reference marginal distribution, with non-selected coordinates left unchanged. Framework name: TraceRouter++.

    Problem formulation

    • Let the activation vector be h ∈ R^d, and let R be the set of routing feature indices that are sensitive to intervention and causally connected to harmful semantic propagation; find a transport map T that minimizes transport cost subject to T# pX = pY and coordinates outside R unchanged (Eq. 1).
    • Full multivariate OT is expensive to compute, so the authors use a coordinate-wise marginal OT approximation: the scalar activation v at coordinate k is aligned in marginal distribution via the empirical quantile map Tk(v) = FYk−1(FXk(v)) (Eq. 7).
    • Proposition 1: under coordinate factorization and separable cost assumptions, the product of coordinate-wise maps is the joint optimal transport map; the authors explicitly state that SAE features are not assumed to satisfy factorization, and the approximation has no general joint alignment guarantee. Proposition 2 gives the minimum-perturbation property under exact joint transport, and the authors note it does not apply to the coordinate-wise implementation; utility preservation relies on empirical evaluation.

    Discover: sensitive routing source identification (III-B, Algorithm 1)

    • Define sensitive attention SA: the head-averaged attention from sensitive modifier tokens to target entity noun tokens (Eq. 2); subtract the contextual disturbance CD on irrelevant tokens (Eq. 3) to obtain the sensitivity score SS(l) (Eq. 4).
    • The layer at the first significant local peak of SS(l) is taken as the sensitive starting layer l* (Figure 3); for Stable Diffusion 1.4, attention is taken from the 12 blocks of the frozen CLIP ViT-L/14 text encoder.
    • At l*, dense activations are decomposed with a Top-K SAE, ranked by ΔWFS = WFS_sens − WFS_benign (Eq. 5), and the Top-K are taken as sensitive routing features.

    Trace: causal semantic routing tracing (III-C, Algorithm 2)

    • The SAE decoder matrix Wdec is used to project sensitive features back into the l* dense space, and the coordinate with the largest projection magnitude is taken as the source routing node Ssrc.
    • A single selective zeroing ablation is performed on Ssrc, caching activations at each layer before and after ablation; the authors state that zeroing ablation is a tractable approximation of causal mediation analysis.
    • The feature influence score FIS(m) = f_dense(m) · µ_dense(m) · ∆ablation(m) (Eq. 6) quantifies downstream node causal influence; Top-K high-FIS nodes are taken layer by layer to reconstruct the causal routing path and construct a binary routing mask M.

    Realign: OT soft routing realignment (III-D, Algorithm 3)

    • For each coordinate selected by the mask, empirical CDFs are estimated from activation samples under unsafe inputs and safe reference inputs, and quantile mapping is performed per Eq. 7; unselected coordinates use the identity map.
    • Realignment is applied coordinate-wise independently as h̃ = h + M ⊙ (T(h) − h) (Eq. 8); at inference time this masked realignment is applied to every input, without relying on an external safety detector.

    Implementation and evaluation setup

    • SAE expansion factor 4×, Adam optimizer, learning rate 4×10⁻⁴, batch size 4096, MSE reconstruction loss; K is determined by the elbow point of the WFS ranking curve; the default OT intervention strength is λ = 1.
    • Safety judgment: Claude 3.5 Sonnet and GPT-4 independently judge DEFENDED/UNSAFE, and on disagreement or UNCERTAIN, three annotators vote by majority; in the image domain, NudeNet detects nudity and violence is evaluated with Q16 per the Ring-A-Bell protocol; main results are repeated over 5 seeds, run on a single NVIDIA A6000.
  4. What experiments did the paper conduct?

    Across three types of architectures, DSR reaches up to 99.5±0.3%, with utility degradation ≤0.6 percentage points; under adaptive GCG it drops markedly.

    Experimental setup

    • Models tested: LLaMA3-8B-Instruct, Mistral-7B-Instruct (LLMs); LLaVA-1.5-7B, MiniGPT-4-7B, DeepSeek-VL2 (MLLMs); Stable Diffusion 1.4, FLUX.1-Dev, AltDiffusion, Show-o2 (DMs).
    • Datasets and attacks: For LLMs, AdvBench and HarmBench are used to construct harmful instructions, with attacks GCG, AutoDAN, Template, Prefill, and combinations; for MLLMs, JailbreakV-28K (T/P/L three text jailbreak types × Noise/SD/Nature/Blank visual contexts) and FigStep (10 safety categories); for DMs, I2P-N, I2P-V, P4D, and Ring-A-Bell.
    • Utility benchmarks: XSTEST (ORR), Global-MMLU-Lite, MMBench, MS COCO (CLIP Score, FID).
    • Metrics: The main metric is DSR (defense success rate); lower ORR is better; lower FID is better.
    • Judging method: Claude 3.5 Sonnet and GPT-4 judge independently, and on disagreement or UNCERTAIN, three annotators vote by majority; images use NudeNet and Q16.
    • Number of repetitions: Main results in Tables II–VII and ablations in Tables VIII–XII are mean ± sample standard deviation over 5 seeds; other results retain their original reporting format.

    Main results

    According to Table II (LLM, DSR%) and Table V (DM, DSR%):

    Swipe horizontally to read the table

    ModelMethodGCGAutoDANTemplatePrefillT+PAvg.
    LLaMA3-8B-InstructOriginal88879841263.2
    LLaMA3-8B-InstructHumorReject9899991009898.8
    LLaMA3-8B-InstructTraceRouter++99.0±0.799.8±0.499.6±0.599.8±0.499.4±0.599.5±0.3
    Mistral-7B-InstructOriginal4222146.6
    Mistral-7B-InstructHumorReject959796989796.6
    Mistral-7B-InstructTraceRouter++99.4±0.599.0±0.798.8±0.498.6±0.598.4±0.598.8±0.3
    • The authors state that TraceRouter++ maintains high DSR across all attack categories, with standard deviation over 5 seeds not exceeding 0.7 percentage points; LEACE achieves an average DSR of only 69.0% (LLaMA3) and 21.2% (Mistral).
    • MLLMs (Tables III, IV): LLaVA-1.5-7B's average DSR on JailbreakV-28K rises from 48.3% to 98.8±0.1%, and is 99.4±0.1% on FigStep; MiniGPT-4-7B is 96.7±0.4% and 99.6±0.1%, respectively; DeepSeek-VL2 is 94.1±0.8% and 96.0±0.5%, which the authors state exceed the reported values of SafePTR and Immune.
    • DMs (Table V): On Stable Diffusion 1.4, I2P-N 99.3±0.2%, I2P-V 96.1±0.4%, P4D 76.0±1.3%, Ring-A-Bell 98.0±1.1%; the authors state the averages exceed the reported values of SNCE and RECE, while noting that the defense rate under P4D is relatively low and vulnerability remains.

    Utility and over-refusal

    • General utility (Table VI): Global-MMLU-Lite accuracy drops by 0.4 percentage points for LLaMA3 and 0.6 for Mistral (71.1±0.4%, 62.4±0.3%); MMBench drops by 0.1, 0.5, and 0.1 percentage points for the three MLLMs; Stable Diffusion 1.4's CLIP Score drops by 0.26 (31.08±0.07), FID 8.62±0.04. The authors state these numbers support a favorable safety–utility trade-off but do not prove downstream benign computation is unchanged; MiniGPT-4's original MMBench is only 16.0%, and the paper notes that its utility evidence considered alone is of limited strength.
    • Over-refusal (Table VII): On XSTEST, LLaMA3 ORR rises from 3.2% to 3.5±0.3%, and Mistral from 1.2% to 2.3±0.3%; comparison methods DeepAug ORR is 39.6% and 62.0%, DeRTa 27.6% and 74.4%.
    • Pixel-level differences (Figure 7): The authors state that baseline methods have broad residual distributions under both benign and sensitive prompts, whereas TraceRouter++ residuals concentrate in sensitive regions with lower perturbation magnitude.

    Mechanism validation and adaptive attacks

    • Counterfactual intervention (Table VIII): Amplifying the identified route raises DSR from 63.2% to 75.8±1.7% (LLaMA3) and from 49.0% to 78.6±0.7% (LLaVA); suppressing the same path lowers DSR to 61.4±1.4% and 18.2±1.9%; for Stable Diffusion 1.4 the effect is reversed, with suppression raising DSR to 98.8±0.4% and amplification lowering it to 73.5±1.3%. The authors state this provides functional evidence that the path participates in safety-related semantic processing.
    • Intervention granularity ablation (Figure 8): The authors state that isolated neuron suppression baselines show large DSR drops under Prefill and P4D attacks, while path-level realignment is higher across architectures; the translated text does not give specific values for this figure.
    • Defense-aware adaptive GCG (Appendix B, Table XVII): The attack scores candidate suffixes against the intervened model; DSR rises from 58.4±1.7% to 73.4±2.1% on LLaMA3 and from 4.8±2.6% to 52.4±1.1% on Mistral; the authors state these are lower than results under non-adaptive attacks, and the difference between the two models indicates that adaptive robustness varies by architecture.

    Other ablations and analyses

    • OT module ablation (Table IX): Under the same path identification, hard intervention DSR is 74.2/77.4/96.3, linear steering 97.4/95.7/72.2 (LLaMA3/LLaVA/SD), and TraceRouter++ 99.5±0.3/99.4±0.1/99.3±0.2, with FID 8.62±0.04 vs. 19.12 and 16.24.
    • Starting layer selection (Table X): first local peak gives DSR 99.5±0.3/99.4±0.1/99.3±0.2 on the three models, global maximum 84.8/78.4/98.3, and fixed threshold 80.8/74.4/96.5.
    • Number of features K (Table XI): At K/2, DSR is 88.6/81.6/96.3; at 2K it does not rise but falls, and FID rises to 12.92±0.05.
    • Random path control (Figure 10): After random path realignment, DSR is 62.8±0.9 (LLaMA3), 51.0±0.8 (LLaVA), 84.7±0.4 (SD), close to the original models.
    • Path scoring ablation (Table XII): Full F·I·S gives DSR 99.4±0.1 on LLaVA, while F or S alone gives only 63.4±0.8 and 66.8±0.6; the authors note that F·I has higher accuracy on LLaMA3 and that some simplified variants have higher CLIP Score.
    • Appendix A: SAE expansion factor 4× is best (Table XIV); as λ increases from 1 to 3, DSR saturates but ORR rises from 3.5±0.3% to 12.4±1.2% (Table XV); inference latency increases by 28.8%, throughput drops by 21.7%, and GPU memory is almost unchanged (Table XVI); under Prefill-type attacks there are failure cases where the attack overrides fixed-strength intervention.
  5. What are some points that can be further explored?

    The authors state that insufficient robustness under adaptive attacks is a future direction; experiments cover 9 models across three types of architectures.

    Limitations and future directions noted by the authors

    • Insufficient adaptive robustness (Appendix B): The authors state that the current method cannot provide general robustness against defense-aware optimization, and that improving cross-model adaptive robustness is an important future direction.
    • Adaptive robustness varies by architecture (Appendix B): The DSR difference between LLaMA3 and Mistral under adaptive GCG (73.4±2.1% vs. 52.4±1.1%) indicates that robustness varies with architecture.
    • Composition of the safety reference set (Appendix A): Sensitivity to reference set size was evaluated only under a fixed construction protocol; independently varying the semantic composition remains an open direction.
    • Strength of utility evidence (IV-D): MiniGPT-4's original MMBench accuracy is only 16.0%, and the paper notes that its utility evidence considered alone is of limited strength, so DeepSeek-VL2 was additionally tested.
    • Residual vulnerability under P4D (IV-B): The authors note that the defense rate under P4D attack is relatively low (76.0±1.3%), indicating that vulnerability remains under this attack.
    • Failure cases (Appendix A): At λ = 1, the forced affirmative prefix of Prefill-type attacks can override fixed-strength intervention to produce unsafe continuations; the authors state this reflects a residual trade-off between over-intervention and under-intervention.

    Experimental coverage

    • 9 models were tested: 2 LLMs, 3 MLLMs, 4 DMs (Table I), among which FLUX.1-Dev, AltDiffusion, and Show-o2 are compared with SNCE only in Table XIII.
    • Attack coverage: on LLMs, GCG, AutoDAN, Template, Prefill, and their combinations, plus defense-aware adaptive GCG; on MLLMs, JailbreakV-28K and FigStep; on DMs, P4D and Ring-A-Bell.
    • Main results and ablations are repeated over 5 seeds; judgments use dual review by Claude 3.5 Sonnet and GPT-4 plus three-person majority voting, and the paper does not report the agreement rate between reviewers and humans.
    • Efficiency measurements (Table XVI) were performed on a single NVIDIA A6000; the paper does not state the model and input scale used for the measurements.
    • The safety reference set is a benign prompt pool constructed by the authors; in the main evaluation all LLMs share the same fixed set (Appendix A).
  6. Summarize the main content of the paper

    Routing-level OT soft realignment defense; DSR up to 99.5±0.3% across three types of architectures, still vulnerable under adaptive attacks.

    TraceRouter++ is an inference-time safety intervention framework for defending against jailbreak and harmful content generation in LLMs, MLLMs, and DMs.

    • Problem: The authors argue that existing interventions are based on a locality assumption, hard-suppressing isolated neurons, so that when harmful semantics propagate along cross-layer routing, they can bypass via alternative paths, and rigid ablation damages general capabilities.
    • Method: In the Discover stage, attention divergence is used to locate sensitive starting layers and Top-K SAE is used to decompose routing features; in the Trace stage, zeroing ablation and FIS are used to reconstruct cross-layer causal paths; in the Realign stage, coordinate-wise marginal OT is used to softly map the unsafe activation distribution toward a safe reference distribution, with non-selected coordinates unchanged.
    • Main results: LLaMA3-8B-Instruct averages DSR 99.5±0.3%, Mistral-7B-Instruct 98.8±0.3% (Table II); LLaVA-1.5-7B achieves 98.8±0.1% on JailbreakV-28K (Table III); Stable Diffusion 1.4 achieves 99.3±0.2% on I2P-N (Table V).
    • Utility cost: Global-MMLU-Lite accuracy drops by no more than 0.6 percentage points, ORR on XSTEST rises from 3.2% to 3.5±0.3% (LLaMA3), and inference latency increases by 28.8%.
    • Exceptions: Under P4D attack, DSR is 76.0±1.3%; under defense-aware adaptive GCG, DSR drops to 73.4±2.1% (LLaMA3) and 52.4±1.1% (Mistral), and the authors state it does not have general adaptive robustness.
    • Authors' conclusion: The authors state that malicious behavior is structurally encoded in distributed cross-layer routing paths, that continuous distribution alignment along causal paths better balances robustness and utility than local suppression, and they position causal routing trajectories as an interface for precise model intervention.

Related papers

Show 3 more papers
Read originalIEEE Transactions on Pattern Analysis and Machine Intelligence 2026