Where in multimodal LM-derived models can Rank-1 DiM refusal be intervened on?
Where Is Rank-1 DiM Refusal Intervenable in Multimodal LM Derivatives?
Translation in progress — untranslated text is omitted.
Under a unified mean and rank-1 DiM extraction plus intervention protocol, the paper locates intervenable sites for refusal in multimodal LM-derived models, finding that under this recipe intervention accessibility is concentrated in the text decoder T, while the perception encoder, projection layer, and fusion pathway (E/P/F) do not provide equivalent effects. In mixed-input settings such as HADES adversarial images paired with harmful text prompts and speech AdvBench presented via TTS or silence, DiM intervention at the text site increases attack success rates, whereas applying the same intervention at E/P/F is ineffective, and attacks follow the harmful text rather than the image or audio content. margin-rank and leverage probes show that this recipe is extreme at T and near chance level at P/E; on VLGuard Unsafe prompts (n=200, judged by Llama-Guard) the direction is consistent, the absolute attack success rate is not high, and at T it rises from 0.030 to 0.150.
- Open-source model repositories release a large number of derivative models (fine-tunes, VLMs, audio wrappers, etc.), but residual-stream safety audits usually intervene only at a single point in the text decoder; the authors ask: under the same DiM protocol, does the attack uplift also exist in the perceptual encoder, projector, and fusion path.
- Using a unified mean/rank-1 DiM extract=intervene protocol (with matched-norm random controls), extract directions and ablate separately at the text decoder T and at non-text sites E/P/F; combine with margin-rank and leverage probes and a Class×site ledger to audit each derivative class.
- The authors report: under HADES mixed image-text input, Text DiM@T raises ASR from 0.034 to 0.616, while non-text DiM at E/P/F shows no gain (0.021→0.021); the @T direction ranks 50/50 among 50 random probes, with P/E close to random; under the generative U-Net, block erasure is effective and mid-block is ineffective.
- The authors state that conclusions are limited to the tested mean/rank-1 DiM recipes and do not rule out other feature families being effective at non-text sites; B→M transfer is mainly on Llama-3.1-8B-Instruct derivatives; mechanism and generative erasure cells often use smaller n; other extraction families such as multi-direction/SAE are left for future work.