Skip to content
Original

MARS: Multimodal safety steering via a textual refusal direction

Harnessing Textual Refusal Directions for Multimodal Safety

Hugging Face Daily Papers· arXiv:2606.31876· NeurIPS 2026· Moreno D'Incà, Nicu Sebe, Massimiliano ManciniUniversity of TrentoIndexed by this site · 23 days ago· Originally publishedDefenses
#Defenses and guardrails

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

MARS steers five MLLMs with the text refusal direction; Qwen3-VL's video jailbreak refusal rate goes from 28.8% to 88.2%.

Threat modelSafety is defined as refusing unsafe queries and correctly answering safe queries. The intervention reads the hidden state of the last token of the input at each decoder layer, steers along the text refusal direction at inference time, and changes only the first generated token. It faces image, video jailbreak, and cross-modal combined unsafe requests. The paper does not use black-box/white-box.

Questions
Multimodal unsafe data is hard to collect, and each modality must be re-collected, while dissemination also raises ethical and legal issues. The authors examine whether the text refusal direction in the LLM backbone alone can be used to improve MLLM refusal of images and videos without retraining.
Methods
MARS extracts a refusal direction from the difference in acceptance and refusal activations on plain text, recenters it using the mean activation of randomly colored images, adaptively steers only states positively aligned with that direction within the text centroid trust region, and uses plain-text scores to select a layer. At inference time it steers only the first generated token, with no training and no multimodal safety data.
Experiments and results
MARS improves image and video refusal on five models. Qwen3-VL's MARS refusal rate on VideoSafetyBench is 88.2% (Table 6). MMMU for four models is essentially on par with zero-shot, and Qwen3.5 drops from 50.8% to 38.2% (Table 4).
Limitations and future work
The authors note that performance may degrade when the refusal direction is weak or absent, that linear centering may not hold in a highly nonlinear space, and that residual over-refusal remains to be reduced. Experiments include five main models plus LLaVA 1.5 in the appendix; the paper does not report the number of repetitions or agreement with human annotation.
Experimental setupQwen3-VL 8B、Qwen3.5 9B et al. · ViSU、HADES et al. · Refusal Rate、Safety score et al.
Evaluated models
Qwen3-VL 8BQwen3.5 9BInternVL 3.5 8BGemma3 4BMolmo2 8BLLaVA 1.5 13B
Benchmarks
ViSUHADESSIUOHoliSafeMMMUMMMUProVideoSafetyBenchMMSafetyBenchVLGuardMMStarAlpacaMaliciousInstructHarmBench
Metrics
Refusal RateSafety scoreAccuracy
AI summaryMARS is a training-free method that requires no multimodal safety data and uses only a textual refusal direction extracted from the LLM backbone to inject safety steering into multimodal large models. Preliminary results show that the textual refusal direction can generalize across image and video modalities; however, its effectiveness is affected by layer selection, steering strength, and cross-modal alignment, and cross-modal misalignment can incorrectly push safe multimodal inputs toward refusal. MARS corrects modality misalignment via activation centering, adaptively adjusts steering strength within a geometrically defined trust region, and selects the best intervention layer, applying the intervention when generating the first token. Evaluated on five SOTA MLLMs for safety, utility, and video jailbreak, safety gains are consistent, and utility is maintained until over-refusal occurs.

MARS is a training-free method that requires no multimodal safety data and uses only a textual refusal direction extracted from the LLM backbone to inject safety steering into multimodal large models. Preliminary results show that the textual refusal direction can generalize across image and video modalities; however, its effectiveness is affected by layer selection, steering strength, and cross-modal alignment, and cross-modal misalignment can incorrectly push safe multimodal inputs toward refusal. MARS corrects modality misalignment via activation centering, adaptively adjusts steering strength within a geometrically defined trust region, and selects the best intervention layer, applying the intervention when generating the first token. Evaluated on five SOTA MLLMs for safety, utility, and video jailbreak, safety gains are consistent, and utility is maintained until over-refusal occurs.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors examine whether MLLM safety can be improved using only the text refusal direction, without collecting multimodal unsafe data and without retraining.

    Can the text refusal direction in the LLM backbone improve MLLM refusal of images and videos without collecting multimodal unsafe data and without retraining.

    • Scenario: The authors consider that MLLM refusal comes from post-training of the LLM backbone, not innately. Multimodal unsafe data must capture cross-modal interactions, and every new modality (e.g., video) requires re-collection; collection and distribution also raise ethical and legal issues, such as non-consensual or child sexual abuse imagery.
    • Existing shortcomings: The authors state that the fragility of adversarial prompts, narrow fine-tuning, and evasion strategies is amplified in multimodal settings, and visual complexity further weakens safety guarantees. Existing refusal-direction extraction still relies on unsafe data, which is harder to obtain for MLLMs.
    • Observation: The authors examine whether the plain-text refusal direction can transfer to images and videos. Preliminary results suggest it can transfer, but it depends on the layer, steering strength, and cross-modal alignment; the latter causes safe multimodal inputs to be wrongly pushed toward refusal.
    • Proposed method: training-free MARS (Modality-Agnostic Refusal Steering): activation recentering, adaptive strength scaling within a geometric trust region, and selection of the intervention layer, operating only on the first generated token.
    • Threat model:
      • Goal: safety means refusing unsafe queries and correctly answering safe queries; ideally the expected refusal is 0 on the safe set and 1 on the unsafe set (Sec. 2.1).
      • Capability: reads the hidden state of the last token of the input at each decoder layer (denoted <assistant> in the paper), and intervenes along the text refusal direction at inference time; the update is added to all token positions, and decoding changes only the first generated token. The paper does not state black-box or white-box.
      • System: an MLLM composed of a vision encoder, a projection module, and a decoder-only LLM; visual tokens and prompt tokens form the multimodal input.
      • Determination: refusal rate is the proportion of responses hitting the predefined template set (templates in Appendix D); HADES additionally uses LlamaGuard 3 8B to count the proportion judged safe.

  2. What related research exists?

    The authors position this paper between LLM activation steering and MLLM safety work, and compare against four baselines.

    The introduction and Sec. 6 divide related work into LLM activation steering and MLLM safety, and the experiments additionally set up four baselines.

    LLM activation steering

    • Refusal direction: Arditi et al. (2024) state that refusal in language models is mediated by a single direction, and that suppressing this direction removes refusal and can be used for jailbreaking. The authors follow its extraction method and state that strengthening refusal in an input-dependent way had not been explored before.
    • Safety subspace and inference-time intervention: Zhao et al. (2025) state that harmfulness and refusal belong to different subspaces; Qi et al. (2025) state that existing alignment often covers only a few tokens. Li et al. (2023) use inference-time intervention to elicit more truthful answers, and Lee et al. (2025) use conditional activation steering to program refusal.
    • Other steering applications: The paper mentions that steering is also used for personality (Chen et al. 2025), emotion (Turner et al. 2024), unsupervised latent knowledge (Burns et al. 2023), and activation-scaling interpretation (Stoehr et al. 2024), as well as COSMIC (Siu et al. 2025). AdaSteer (Zhao et al. 2025) uses activation steering to improve safety, and the authors use it as a reference for transferring plain-text steering to multimodal.

    MLLM safety

    • Training-based realignment: Chen et al. (2024)'s DRESS aligns vision-language models with natural language feedback; Li et al. (2024) red-team vision-language models. The authors state that such methods are effective but annotation is expensive, they remain vulnerable to jailbreaking, and unsafe multimodal data must be collected and distributed; Cintaqia et al. (2025) and Kale et al. (2026) are cited as positions against such collection.
    • Representation-space methods: Wang et al. (2025) suppress harmful features using directions derived from adversarial images; SASA (Wang et al. 2025) trains a lightweight classifier that replaces outputs labeled unsafe with a fixed refusal; AutoSteer (Wu et al. 2025) trains a safety detector that activates a learned refusal head after detecting harmful input.
    • Prompting methods without multimodal annotation: ECSO (Gou et al. 2024) has the model self-evaluate and regenerate, requiring no multimodal safety data but requiring multiple forward passes.

    Baselines and benchmarks

    • Baseline methods: AdaSteer, ECSO, AutoSteer, SASA. The authors state that SASA is trained on multimodal, utility, and jailbreak data and is used as an upper bound; AutoSteer likewise uses multimodal safety data.
    • Benchmarks/datasets: Main experiments are ViSU, HADES, SIUO, HoliSafe, MMMU, MMMUPro, VideoSafetyBench; the appendix covers MMSafetyBench, VLGuard, MMStar. Direction extraction uses Alpaca and MaliciousInstruct, and layer-selection validation uses HarmBench.

    The authors state that the difference between this paper and these works is: strengthening and transferring the refusal direction from text, rather than suppressing it to jailbreak, and not training a multimodal safety classifier. The authors state that this is the first analysis of the text refusal direction in multimodal models, and that text data is easier to collect and does not require propagating unsafe multimodal data.

  3. How does the paper address this problem?

    MARS recenters activations, adaptively steers within a trust region, and uses plain-text scores to select the intervention layer.

    MARS extracts the refusal direction from the accept/refuse activations on plain text of the LLM backbone, and adds it to the multimodal hidden states at inference time. Recentering removes the modality component unrelated to safety, a ReLU gate decides whether to steer, the trust-region radius decides the strength, and plain-text scores select the layer. No training, no multimodal safety data.

    Problem setting and refusal direction

    • An MLLM is denoted as a function from text and visual inputs to text outputs, composed of a vision encoder, a projection module, and a decoder-only LLM. Safety uses a binary function g: g=1 if the response is a refusal.
    • The direction is taken from the hidden state of the last input token at each decoder layer. The safe text set is Alpaca, the unsafe text set is MaliciousInstruct; responses that do not correspond to acceptance or refusal are filtered out, yielding an acceptance set and a refusal set.
    • The refusal direction at each layer is the refusal mean minus the acceptance mean, dℓr=μℓref−μℓacc, then divided by the L2 norm. Its meaning is the average activation difference of refusal relative to acceptance at that layer.
    • Preliminary steering adds to the hidden state the strength α multiplied by its projection onto that unit direction: same direction strengthens refusal, opposite direction strengthens acceptance. Appendix D fixes 100 entries per class and specifies deterministic decoding.

    Modality misalignment and recentering

    • Sec. 3 uses ViSU images for description on Qwen3-VL 8B, Gemma3 4B, and InternVL 3.5 8B, examining layer and α; utility is measured on MMMUPro. The authors state that on most layers unsafe inputs are refused more often, but the same layer also over-refuses safe inputs, and the projection fails to isolate safety features on its own.
    • The authors use a separation score Sep_ℓ to compare the average cosine of unsafe versus safe image activations relative to the refusal direction. The Fig. 2 caption states that in the original space both classes of images project toward refusal; after centering, unsafe aligns with refusal and safe aligns with acceptance. The authors state that the <assistant> token is dominated by a modality component aligned with the refusal direction.
    • Recentering does not use safe/unsafe multimodal data: it takes the mean activation of N randomly colored images at that layer and subtracts it from the original activation. The main text does not give N for the main experiments; Appendix Table 13 ablates n=0, 8, 24, 48, 64, 128.

    Gating, trust region, and layer selection

    • At inference, the neutral mean is subtracted first, then the projection onto the refusal direction is taken and passed through ReLU. Updates with negative alignment (which the authors estimate as safe) are 0, with no separately trained classifier. Only positive alignment yields nonzero steering.
    • The trust-region radius is the L2 distance from the centered activation to the nearest acceptance or refusal text centroid. The steering coefficient is αℓ=rℓ/(|sℓ|2+ϵ), where ε is a small positive number for numerical stability; the paper does not give its specific value. Its meaning is to limit the displacement to the region covered by the text centroids.
    • The update is added to all token positions; decoding only steers the first generated token. The authors state that this token largely determines the subsequent generation trajectory.
    • Layer selection uses only text. Three scores: direction consistency (cosine with the validation-set refusal direction), context separability (the average of nearest-centroid classification accuracy), and safe-text margin (the projection margin of safe validation activations relative to the midpoint of acceptance/refusal). After min-max normalization across layers they are summed, and the argmax is taken. Appendix D: unsafe validation uses 100 entries per class from HarmBench, safe validation uses another 100 Alpaca entries. The selected layers are 26 for Gemma3 4B, 22 for Molmo2 8B, 23 for Qwen3-VL 8B, 22 for Qwen3.5 9B, and 25 for InternVL 3.5 8B.

    Success determination

    • Refusal rate: the proportion of responses matching the predefined template set. Appendix D lists the template phrases; they are not reproduced here.
    • Safety score: the proportion of responses judged safe by LlamaGuard 3 8B. This model processes text only; on HADES the unsafe prompt is passed in as image context.
    • Utility: the accuracy native to each benchmark.

  4. What experiments did the paper conduct?

    On five MLLMs, MARS improves refusal of unsafe content; utility is on par for most, and Qwen3.5 shows over-refusal.

    Experimental setup

    • Models: Qwen3-VL 8B, Qwen3.5 9B, InternVL 3.5 8B, Gemma3 4B, Molmo2 8B. Appendix G additionally tests LLaVA 1.5 13B. The preliminary analysis in Sec. 3 uses only Qwen3-VL, Gemma3, InternVL 3.5.
    • Data scale: ViSU is 5K safe–unsafe image pairs, 20 classes; HADES is 4.5K image-text pairs, 5 classes of visual jailbreak; SIUO is 167 cases, 9 domains; the SSU split of HoliSafe is 718 entries, 7 domains; MMMU is 11.5K questions, 30 subjects, 183 subfields; MMMUPro is 5.2K; VideoSafetyBench is 2,264 video–text pairs, 13 classes.
    • Appendix data: MMSafetyBench totals 6,720 entries (5,040 image-text plus 1,680 plain text, 13 classes); VLGuard is about 1K unsafe samples, split into US and UU; MMStar is 1,500 multiple-choice questions that require looking at the image.
    • Baselines: training-free ECSO, AdaSteer; training-based AutoSteer, SASA (marked in red in the tables). The authors state that SASA is used as an upper bound.
    • Metrics and judging: refusal rate uses template matching; HADES's safety score is computed by LlamaGuard 3 8B; utility is accuracy. The paper does not report the number of repetitions, nor the agreement of template judgment or LlamaGuard with human annotation.
    • Protocol: Appendix D specifies deterministic decoding, a single NVIDIA A100, FP16 automatic mixed precision. The authors state that runtime is roughly equal to original model inference, ranging from 30 minutes to several hours depending on model and dataset. The Zero-shot and MARS refusal rates in Table 2 are respectively the same as Neutral Prompt and MARS (Neutral Prompt) in Appendix Table 8; the Table 2 title does not state the prompt.

    Main results

    Refusal rate (%). ViSU from Table 1, HADES from Table 2, video from Table 6. Zero-shot in parentheses.

    Swipe horizontally to read the table

    ModelViSU unsafeHADES refusalVideo refusal
    Gemma3 4B43.3% (18.2%)59.2% (17.9%)not reported
    Molmo2 8B68.1% (0.2%)95.8% (17.2%)67.8% (28.8%)
    InternVL 3.5 8B70.4% (11.2%)88.3% (16.7%)11.9% (4.7%)
    Qwen3-VL 8B78.5% (14.2%)98.3% (62.0%)88.2% (28.8%)
    Qwen3.5 9B51.1% (22.9%)99.5% (65.3%)78.2% (26.2%)
    • The authors state that the improvement in ViSU unsafe refusal ranges from +25.1% for Gemma3 to +67.9% for Molmo2, with over-refusal lower than the original space in Fig. 1b. The authors state that unsafe refusal is higher than AutoSteer for all five models; relative to SASA, they claim higher only for Molmo2, Qwen3-VL, InternVL 3.5. In Table 1, Gemma3 and Qwen3.5 remain below SASA (43.3% vs 90.0%, 51.1% vs 65.1%), and SASA pushes safe-image refusal on Gemma3 to 71.1%.
    • On HADES the authors state that all models recover refusal, with Molmo2 highest at +78.6%, and that on Molmo2, Qwen3-VL, Qwen3.5 it is comparable to SASA. The corresponding refusals in Table 2 are 95.8% vs 98.4%, 98.3% vs 100.0%, 99.5% vs 99.9%; Gemma3 is 59.2%, below AutoSteer's 92.1% and SASA's 100.0%. InternVL 3.5's MARS safety score is 79.8%, below AutoSteer and SASA's 95.9%.
    • On video the authors state that MARS has the highest refusal rate among training-free methods, Qwen3-VL is +59.4% relative to zero-shot, and it is higher than that model's SASA (which the authors state is +47.9%). Table 6 marks InternVL 3.5 as ×2.5; in that row SASA is 76.9% and AutoSteer is 30.0%. Gemma3 is not in that table.

    Cross-modal safety

    • In Table 3, image and text are each safe, and the combination is unsafe. Zero-shot ranges from 3.6% for InternVL 3.5 on SIUO to 40.0% for Qwen3.5 on HoliSafe SSU. The authors state that ECSO's improvement is limited and sometimes decreases; in Table 1 the authors state that ECSO fails except for Molmo2 (+36.8%), but InternVL 3.5's unsafe refusal goes from 11.2% to 16.1%.
    • MARS on SIUO is Gemma3 12.0%, Molmo2 80.8%, InternVL 3.5 29.9%, Qwen3-VL 76.7%, Qwen3.5 58.7% (zero-shot 6.0%, 22.8%, 3.6%, 20.4%, 27.5%). The authors state that Molmo2 is highest on SIUO at +58%, and that except for Gemma3 it is higher than AutoSteer on both datasets. In Table 3, Gemma3's HoliSafe SSU is MARS 19.9%, AutoSteer 25.6%. The authors state that MARS is higher than SASA on SIUO for Molmo2 and Qwen3-VL (80.8% vs 64.1%, 76.7% vs 65.9%).
    • Appendix Table 10 full HoliSafe: MARS is 29.0%, 84.1%, 55.2%, 80.5%, 75.6% (zero-shot 27.8%, 41.5%, 18.9%, 37.2%, 54.2%). The authors state that on Gemma3 all training-free methods struggle to improve (AutoSteer is 44.2%). In Appendix Table 9, MARS's US/UU averages are 57.5%, 85.4%, 82.4%, 89.8%, 85.5%, higher than AutoSteer in the same table and mostly lower than SASA.

    Utility and component ablations

    • Table 4: MARS on Gemma3, Molmo2, InternVL 3.5, Qwen3-VL is the same as zero-shot or 0.1 percentage point lower (Gemma3's MMMU is 39.8% in both). Qwen3.5 drops from 50.8%/35.6% to 38.2%/24.5%. SASA on InternVL 3.5's MMMU is 41.3% (zero-shot 51.3%), and Qwen3.5 is 30.8%/22.5%. The authors state that Qwen3.5 has moderate over-refusal but is still more controlled than SASA. On MMStar in Appendix Table 11, MARS matches zero-shot for the first four models, and Qwen3.5 goes from 39.1% to 35.4%; the authors state that this model occasionally starts with a refusal instead of an option letter.
    • Table 5 and Table 12: after removing layer selection, Gemma3 and Molmo2's unsafe refusal drops to 0.8% and 0.6%, Qwen3-VL's unsafe refusal is 0.0% and safe refusal is 54.0%, and InternVL 3.5's MMMUPro goes from 36.5% to 18.0%. After removing centering, Molmo2's safe refusal is 99.2%. After removing ReLU, Gemma3's MMMUPro is 24.9% and InternVL 3.5's is 34.6%. The authors state that layer selection preserves integrity, centering separates the refusal signal, and ReLU limits the intervention to unsafe activations.
    • Appendix G: LLaVA 1.5 refuses only 33% of MaliciousInstruct when extracting the direction, and the direction comes from 33 refusal samples. The main text states HADES improves by 32.3% and ViSU by 28.1%, while maintaining acceptance of safe inputs. Figure 4 is labeled 32.3 (×1615) and 28.2 (×352).

    Other ablations and analyses

    • Removing direction consistency (Table 12): Molmo2's unsafe refusal is 57.7% (MARS is 68.1%), InternVL 3.5's safe refusal is 21.5%; after removing separability, Molmo2's unsafe refusal is 12.3% and Gemma3's is 36.3%. For the two Qwen rows the numbers are unchanged after removing a single item, and the authors state that the remaining components still select the same layer.
    • Number of neutral images (Table 13): at n=0 Molmo2's safe refusal is 99.2%; at n=8 its unsafe refusal is 63.7%; the n=128 row is the same as the MARS row in Table 1. The authors state that the estimate is stable with respect to sample size, and that Molmo2 has a small decrease at n=8 and n=24.
    • Appendix Table 8: under the original unsafe prompt, refusal is 85.7%, 89.4%, 76.6%, 94.7%, 96.5%; under the semantically neutral prompt it is 17.9%, 17.2%, 16.7%, 62.0%, 65.3%. The authors state that the drop ranges from -31.2% for Qwen3.5 to -72.2% for Molmo2. The prompt is not reproduced.
    • Appendix Table 7, MMSafetyBench refusal: MARS is 32.5%, 88.3%, 55.9%, 85.3%, 76.9% (zero-shot 23.7%, 39.8%, 26.3%, 46.3%, 45.2%). SASA is 99.9% to 98.4%; a footnote states that SASA was trained on jailbreak data.
    • Fig. 1 main text: when α is increased to 3.0, unsafe refusal rises, safe acceptance falls, and utility falls; Gemma3's safety performance declines at α=2.0. The peak unsafe-refusal layers are 31 for Gemma3, 16 for Qwen3-VL, and 19 for InternVL 3.5; Gemma3's layer 10 refuses 96.4% of safe queries at α=1.0.
    • Fig. 3: the authors state that zero-shot Molmo2 gives step-by-step guidance for harmful requests, and MARS steers it to refusal. Appendix H has more qualitative examples; prompts and outputs are not reproduced here.

  5. What are some points that can be further explored?

    The authors point out weak refusal signals, the linear-centering approximation, and residual over-refusal on safe inputs.

    Limitations and future directions noted by the authors

    • Weak or missing refusal direction: MARS assumes the refusal direction exists in the LLM backbone; performance may degrade when the signal is weak or missing. The authors state they tested weak alignment on LLaVA (Limitations, Appendix G).
    • Linear centering is an approximation: The authors assume modality misalignment can be mitigated by linear centering, and state that this may not hold in a highly nonlinear activation space (Limitations).
    • Residual over-refusal: Residual over-refusal remains on safe inputs. The authors list reducing over-refusal while preserving safety gains as future work (Limitations).

    Scope of experiments

    • The main experimental models are Qwen3-VL 8B, Qwen3.5 9B, InternVL 3.5 8B, Gemma3 4B, and Molmo2 8B, 5 in total; Appendix G additionally includes LLaVA 1.5 13B (Sec. 5.1, Appendix G).
    • Safety evaluation includes ViSU, HADES, SIUO, HoliSafe (SSU and full), VideoSafetyBench, MMSafetyBench, and VLGuard; utility includes MMMU, MMMUPro, and MMStar (Sec. 5, Appendices A–B). The video models in Table 6 do not include Gemma3.
    • Comparison methods are ECSO, AdaSteer, AutoSteer, and SASA (Sec. 5.1). Safety scores use LlamaGuard 3 8B; the paper states that this model is text-only, and on HADES an unsafe prompt is additionally passed in (Sec. 5.1, Appendix D).
    • The refusal direction is extracted using 100 examples each from Alpaca and MaliciousInstruct; layer selection validation uses 100 examples per category from HarmBench and another 100 from Alpaca (Appendix D). The selected layer is written in Appendix D.
    • The paper does not report the number of repetitions, nor the agreement of template judgments or LlamaGuard with human judgment; the main text does not state the number of neutral images in the main experiments (Appendix Table 13 ablates n separately).

  6. Summarize the main content of the paper

    The text refusal direction, via recentering and adaptive steering, can transfer to images and videos, but there are over-refusal exceptions.

    MARS is a training-free inference-time intervention that uses a pure-text refusal direction to strengthen MLLM refusal of images and videos.

    • Problem: The authors argue that multimodal unsafe data is hard to collect and unsuitable to disseminate, while MLLMs mainly inherit safety from the language backbone and refuse insufficiently when facing images, video jailbreaks, and cross-modal combined requests.
    • Method: Extract a unit refusal direction from the accept/refuse activation difference between Alpaca and MaliciousInstruct; remove the modality component using the mean over randomly colored images; adaptively add steering within the text centroid trust region only to activations positively aligned with that direction; select a layer using pure-text scores, and steer only the first generated token.
    • ViSU: On unsafe images, MARS refusal is Gemma3 43.3%, Molmo2 68.1%, InternVL 3.5 70.4%, Qwen3-VL 78.5%, Qwen3.5 51.1% (Table 1; zero-shot 18.2%, 0.2%, 11.2%, 14.2%, 22.9%). Refusal on safe images is 1.3%, 7.3%, 10.3%, 2.2%, 0.6%.
    • Jailbreak: On HADES, MARS refusal is 59.2%, 95.8%, 88.3%, 98.3%, 99.5% (Table 2). On VideoSafetyBench, Molmo2 67.8%, InternVL 3.5 11.9%, Qwen3-VL 88.2%, Qwen3.5 78.2% (Table 6). The authors state Qwen3-VL is +59.4% relative to zero-shot. Gemma3's HADES refusal of 59.2% remains below SASA's 100.0%.
    • Utility and failure modes: MMMU for four models is essentially on par with zero-shot; Qwen3.5 drops from 50.8% to 38.2% (Table 4). Without centering, Molmo2's refusal on safe images is 99.2% (Table 5).
    • Authors' conclusion: The authors state that the text refusal direction encodes a safety signal transferable to images and videos, that strengthening it can improve safety without multimodal safety supervision, sometimes comparable to or higher than training-based methods, and that the computational overhead is negligible. The authors also note residual over-refusal, the linear centering assumption, and that performance may degrade when the refusal direction in the backbone is weak.

Related papers

Both use: MMStar
Both use: VLGuard
Both use: MMMU
Show 3 more papers
Both use: MMMU
Both use: HADES
Read originalNeurIPS 2026