Text-Anchored Semantic Perturbation: A Transferable Jailbreak Attack against Multimodal Large Models
Text-Anchored Semantic Perturbations for Transferable Jailbreak Attacks on Multimodal Large Language Models
The authors propose the black-box jailbreak framework TA-SPA, which optimizes transferable perturbations in a text-anchored semantic space and achieves the highest G-ASR in 8 of 9 model-dataset combinations.
Standard black-box setting: the attacker can only submit multimodal inputs and text prompts through an API and observe responses, cannot access model parameters θ or internal safety mechanisms, and cannot modify, remove or introduce any system messages; the attack proceeds only through user-side inputs; the goal is to make the model give harmful or policy-violating content rather than refuse in response to malicious queries.
- MLLM safety alignment is mainly learned in text representation space and is difficult to transfer to fused cross-modal representations; malicious multimodal inputs can bypass refusal. Existing multimodal jailbreak attacks rely on surface visual form or lexical cues, have limited cross-model transferability, and are easily mitigated by input-level defenses.
- Proposes the black-box jailbreak framework TA-SPA: TASF uses a frozen CLIP encoder to decompose image-text representations into cross-modal semantic factors and modality residuals; SPA perturbs the text-side residual while fixing the semantic factor to construct diverse target anchors, then uses a contrastive margin loss to perform PGD optimization of the perturbation in this semantic space and transfers it to the victim model.
- On LLaVA-7B, MiniGPT-4, InstructBLIP and three benchmarks, TA-SPA achieves the highest G-ASR in 8 of 9 model-dataset combinations; transfer to GPT-4o, Gemini, ERNIE averages 52.3% G-ASR, above the strongest baseline IDEATOR's 49.4%; average ASR is highest under four defenses.
- The authors note: validation is only on image-text MLLMs and a fixed set of benchmarks/defenses; performance under other modalities, safety policies and deployment pipelines remains to be verified; automatic metrics may not capture fine-grained harm severity; joint evaluation of representation-layer defenses is left for future work. Evaluation for each setting is based on 40 representative requests.
- Evaluated models
- LLaVA-7BMiniGPT-4InstructBLIPGPT-4oGeminiERNIE
- Benchmarks
- AdvBenchMM-SafetyBenchVAJM
- Metrics
- D-ASRG-ASRToxicity (Google Perspective API)
The researchers propose Text-Anchored Semantic Perturbation Attack (TA-SPA), a black-box jailbreak framework against multimodal large models that optimizes transferable perturbations in a text-anchored semantic space. The method combines Text-Anchored Semantic Factorization (TASF) to separate cross-modal semantic factors from modality-specific residuals; it also combines Semantic-Preserving Augmentation (SPA) to expand harmful target anchors while preserving semantic consistency. Experiments show that the attack is effective, can transfer to commercial multimodal large models, and is competitive under several representative defenses. The authors state that additional control experiments and probe results support this semantic decomposition, but this does not mean full disentanglement has been achieved; this also indicates that beyond input-layer filtering, safety alignment needs to be advanced from the representation layer.
Paper analysis
What problem does this paper attempt to solve?
Addressing the problem that safety alignment of multimodal large models is difficult to transfer from text space to cross-modal representations, the authors propose a transferable black-box jailbreak attack.
Core problem: MLLM safety alignment is learned in text representation space but cannot reliably transfer to fused cross-modal representations, so multimodal inputs can be jailbroken; the authors study how to construct transferable black-box multimodal jailbreak attacks.
- Problem setting and importance: MLLMs are widely deployed in tasks such as multimodal reasoning and visual question answering, but the authors hold that their safety alignment remains fragile, and malicious multimodal inputs can induce harmful or policy-violating responses, affecting real-world deployment safety.
- Shortcomings of existing methods: The authors state that existing attacks (e.g., FigStep, HADES, MML) rely on surface visual form, typographic layout or explicit cross-modal decoding patterns, have limited cross-model transferability, and can be mitigated by defenses such as OCR filtering, visual sanitization and prompt-level screening; optimizing in pixel space or entangled feature space easily overfits to the textures and noise of a specific proxy model.
- Core observation and hypothesis: The authors hypothesize that these limitations relate to the entanglement of high-level semantic intent with low-level modality artifacts; safety mechanisms may rely on surface shortcuts such as lexical cues or modality residuals, and once these cues are removed or redistributed across modalities, harmful intent remains semantically reachable to the model yet invisible to filters.
- Proposed method: Proposes the Text-Anchored Semantic Perturbation Attack (TA-SPA), which optimizes adversarial perturbations in a text-anchored semantic space rather than an entangled multimodal feature space, comprising two components: TASF (semantic-residual decomposition) and SPA (semantic-preserving enhancement).
- Threat model:
- Goal: Construct an adversarial multimodal input xadv so that the model gives actionable or substantive harmful assistance to a malicious query, rather than maintaining refusal.
- Knowledge: Standard black-box setting; the attacker cannot access model parameters θ or internal safety mechanisms.
- Capability: Can only submit multimodal inputs and text prompts via API and observe responses; cannot modify, remove or introduce any system message; the attack proceeds only through user-side inputs.
- Victim systems: Open-source MLLMs (LLaVA-7B, MiniGPT-4, InstructBLIP) and commercial systems (GPT-4o, Gemini, ERNIE).
What related research exists?
Related work falls into three groups: MLLMs and their alignment, multimodal jailbreak attacks, and transfer attacks; this paper compares with 5 multimodal and 5 text attack baselines.
Multimodal large models and safety alignment
- ViECap, LLaVA, etc.—connect a visual encoder with a language model (e.g., GPT-2, Vicuna) via a projection module to achieve image-text understanding and generation; the authors note that the safety robustness of such systems has not improved in step with their capabilities.
- Alignment techniques such as SFT, RLHF, DPO, RLAIF—the authors state that they are mainly designed around text instructions and responses, and when transferred to fused multimodal representations, the model may still be misled by harmful intent encoded in forms not adequately covered by text alignment.
Multimodal jailbreak attacks
- ImgJP (Niu et al., 2024)—demonstrates that adversarial visual perturbations can steer MLLMs to produce unsafe responses, treating the visual channel itself as an attack surface.
- FigStep (Gong et al., 2025)—renders harmful instructions as typographic visual prompts, exploiting visual text recognition to bypass text-centric safeguards.
- HADES (Li et al., 2024b)—hides and amplifies malicious intent with carefully crafted images; IDEATOR (Wang et al., 2025a)—uses the LVLM itself to automatically generate and evaluate jailbreak prompts; MML (Wang et al., 2025b)—disperses malicious intent across the image and text channels, reconstructing harmful instructions through multimodal linking.
Transfer attacks
- Feature-space attacks (Inkawhich et al., 2019) and Intermediate Level Attack (Huang et al., 2019)—improve black-box transferability by optimizing intermediate-layer feature representations; the paper notes that TA-SPA differs in that it decomposes paired image-text representations into text-anchored semantic and modality residual components and optimizes a cross-modal jailbreak objective, rather than attacking undecomposed CNN feature maps.
Baselines and benchmarks
- Multimodal baselines: imgJP, FigStep, HADES, IDEATOR, MML; text baselines: GCG, AutoDAN, PAIR, TAP. Defense evaluation includes VLGuard, ProEAT, AdaShield, BlueSuffix. Benchmarks are AdvBench, MM-SafetyBench, VAJM.
The authors position this paper as: existing attacks rely on modality-specific surface patterns, have limited transferability and are exposed to input-level defenses; TA-SPA instead perturbs text-anchored semantic representations to reduce reliance on surface visual or lexical cues.
How does the paper address this problem?
TA-SPA first uses TASF to decompose semantic and residual factors, then uses SPA to diversify target anchors, and finally performs contrastive PGD optimization in the text-anchored semantic space.
Overall idea: rather than optimizing directly in pixel space or entangled feature space, it uses a public CLIP encoder to build a text-anchored semantic space, optimizes a transferable perturbation within it, and then transfers the adversarial image to the victim MLLM. The framework is named TA-SPA and consists of two components, TASF and SPA.
Problem formulation
- The MLLM defines a conditional distribution pθ(y | x, t); the attacker constructs xadv to maximize the probability of a harmful target response: maxxadv log pθ(yh | xadv, t), with the constraint |xadv − x|∞ ≤ ϵ ensuring the perturbation is quasi-imperceptible.
- Success criterion: the model gives actionable or substantive assistance to the malicious query, rather than refusing.
TASF: text-anchored semantic decomposition
- Uses frozen CLIP image/text encoders Fx, Ft as public proxies to decompose each representation into a semantic factor s and a residual factor u: s captures text-anchored cross-modal invariant propositional content, and u absorbs modality-form variation not needed for cross-modal semantic consistency.
- The training objective is a multi-term loss: Lsim aligns semantic factors of paired image-text, Ldiff suppresses redundancy between semantic and residual channels via soft orthogonality, Lrec ensures the original CLIP features can be reconstructed, and Lclip anchors the learned space to the public CLIP geometry with a symmetric CLIP contrastive objective; the total objective is LTASF = λsimLsim + λdiffLdiff + λrecLrec + λclipLclip.
- Gradients act only on the lightweight decomposition modules {Esx, Eux, Est, Eut, Rx, Rt}, while the CLIP encoders remain frozen; TASF is trained offline on 128 anchored requests, independent of the victim model.
SPA: semantic-preserving enhancement
- Fixes the source image semantic factor ss, applies dropout and Gaussian noise only to the text-side residual ut to obtain perturbed residuals, then recombines them with ss to form a set of source-conditioned anchors Aspa that share a fixed semantic factor but differ in residual realization.
- During optimization these anchors serve as negative samples, pushing the adversarial representation away from the source proxy residual shortcut; the authors state this can reduce reliance on a single lexical trigger. The authors explicitly state that semantic preservation is a design goal, not evidence that the residual is unintentional.
Attack pipeline and optimization
- An auxiliary LLM generates m semantically equivalent rewrites of the malicious query (Appendix A.5 gives the full rewrite prompt, generated with GPT-4, default m = 5); the adversarial image semantic factor sadv is extracted and recombined with the text residual of each rewritten query.
- Contrastive margin loss L(δ) = max(0, γ − Star + β Ssrc): increases alignment of the adversarial representation with multiple rewritten queries (Star) and suppresses its similarity to source-conditioned residual anchors (Ssrc); defaults β = 0.7, γ = 0.3.
- The perturbation is optimized with PGD: ∥δ∥∞ ≤ 8/255, T = 100 steps, step size η = 1; the full pipeline is given in Appendix Algorithm 1.
Evaluation method
- Metrics: dictionary-matching ASR (D-ASR), GPT-4-as-judge G-ASR (binary judgment: actionable harmful assistance and not a refusal scores 1; Appendix A.3 gives the judge prompt), and Google Perspective API toxicity score.
- For text-only benchmarks (AdvBench, VAJM), Stable Diffusion 3 Medium is used to synthesize source visual proxies at 512×512.
What experiments did the paper conduct?
On 3 open-source models × 3 benchmarks, G-ASR is highest in 8/9 combinations; transfer to commercial models averages 52.3%, and average ASR is highest under four defenses.
Experimental setup
- Models tested: open-source LLaVA-7B, MiniGPT-4, InstructBLIP; transfer evaluation on commercial systems GPT-4o, Gemini, ERNIE.
- Datasets: AdvBench (520 harmful requests), MM-SafetyBench (4,680), VAJM (1,225); TASF is trained on 128 requests, and comparison results are reported on 40 representative requests following prior work.
- Baselines: multimodal attacks imgJP, FigStep, HADES, IDEATOR, MML; text attacks GCG, AutoDAN, PAIR, TAP.
- Metrics: D-ASR (dictionary matching), G-ASR (GPT-4 judge), toxicity score (Google Perspective API).
- Proxy and hyperparameters: frozen CLIP ViT-B/32 as proxy encoder; ∥δ∥∞ ≤ 8/255, PGD 100 steps, η = 1, β = 0.7, γ = 0.3, m = 5.
- Number of repetitions: not reported in the paper.
Main results
According to Table 1 (D-ASR / G-ASR / Tox., %):
Swipe horizontally to read the table
Model Method AdvBench G-ASR MM-SafetyBench G-ASR VAJM G-ASR LLaVA-7B Ours 82.0 78.5 78.5 LLaVA-7B Strongest baseline HADES 76.5 75.5 72.0 MiniGPT-4 Ours 75.0 60.0 76.0 MiniGPT-4 Strongest baseline (by column) 75.5 (HADES) 71.0 (HADES) 66.5 (FigStep) InstructBLIP Ours 70.5 80.0 57.0 InstructBLIP Strongest baseline HADES 67.5 78.5 41.0 - The authors state that TA-SPA achieves the highest G-ASR in 8 of 9 model-dataset combinations; the exception is MiniGPT-4 × AdvBench, where HADES's 75.5 is higher than Ours's 75.0 (Table 1).
- The authors note that some baselines (FigStep, IDEATOR, MML) have higher D-ASR but lower G-ASR and toxicity, and hold that they trigger more shallow keyword-level failures; TA-SPA is more balanced across the three metrics.
- For the text attack comparison (Table 2, AdvBench): Ours on MiniGPT-4 has D-ASR/G-ASR/Tox of 80.5/75.0/0.61, and on LLaVA-7B 85.5/82.0/0.61, both exceeding GCG, AutoDAN, PAIR, TAP; the authors hold that text-only jailbreaks are limited by the discrete text space in which safety alignment is mainly imposed.
Transfer to commercial models
- According to Table 3 (MM-SafetyBench, G-ASR): Ours on GPT-4o, Gemini, ERNIE is 52.3, 68.9, 35.7 respectively, averaging 52.3%, above the strongest baseline IDEATOR's average of 49.4% (GPT-4o 45.5, Gemini 64.2, ERNIE 38.6).
- Exception: on ERNIE, IDEATOR's 38.6 is higher than Ours's 35.7; the authors state this may stem from differences in language priors or alignment strategies, and hold that Ours is the only method near-optimal on all three platforms.
Ablation and perturbation control
- According to Table 4 (MiniGPT-4, average G-ASR over three benchmarks): direct CLIP objective matching 18.2%, adding TASF 23.4%, adding target rewriting 43.1%, adding source repulsion 55.6%, full TA-SPA 60.0%.
- According to Table 5 (MiniGPT-4 matched control): clean proxy (δ = 0) already reaches 45.0% G-ASR, random bounded noise 46.0%, full TA-SPA 60.0%; the authors state that under the same proxy, queries, victim model and judge settings, the gain cannot be explained solely by the proxy or arbitrary bounded perturbations.
Defense robustness
- According to Table 9 (G-ASR): under the four defenses VLGuard, ProEAT, AdaShield, BlueSuffix, Ours achieves the highest average ASR and ranks first in 10 of 12 defense-model combinations.
- Exceptions: on MiniGPT-4 under ProEAT, FigStep 28.5 is higher than Ours 27.1; under BlueSuffix, FigStep 28.5 is higher than Ours 28.0; the authors state that Ours still has the best cross-defense average on MiniGPT-4.
Other ablations and analyses
- Decomposition validation (Table 6): linear probing on frozen TASF features gives harm-category macro-F1 of 0.75 on semantic factor s and 0.46 on residual u; the authors state this supports the expected separation but is not evidence of complete disentanglement.
- Proxy architecture (Table 7): with SigLIP and BLIP, ASR is 50.0% and 55.0% respectively, versus 60.0% for CLIP; the authors state that proximity of encoder families may help, but transfer is not explained solely by CLIP-family sharing.
- Judge calibration (Table 8): ASR by human majority vote, Qwen3 and DeepSeek judgments is 77.0%, 65.0%, 68.0% respectively; the authors position G-ASR as an automatic metric rather than a gold-standard human label. The human sample size is not stated in the paper.
- Hyperparameter sensitivity (Appendix A.3): for β ∈ {0.3, 0.5, 0.7, 0.9}, ASR is 72.2%, 75.4%, 76.2%, 75.1%; for m ∈ {1, 3, 5, 7}, ASR is 72.6%, 75.3%, 80.1%, 80.0%, with runtimes of 86s, 130s, 180s, 245s.
- Efficiency (Figure 2): the authors state TA-SPA takes about 120s per sample, with a better ASR-versus-time tradeoff than baselines; Figure 3 is a cross-dataset generalization matrix, and the authors state that AdvBench and VAJM are stronger source domains while MM-SafetyBench has relatively weaker transferability (specific values in the figure cannot all be mapped to rows and columns due to text conversion, so they are not listed).
What are some points that can be further explored?
The authors point out three limitations—modality and benchmark coverage, granularity of automatic metrics, and joint evaluation of representation-layer defenses—and leave them for future work.
Limitations and future directions noted by the authors
- Modality and setting coverage: TA-SPA is mainly validated on image-text MLLMs and a fixed set of benchmarks/defenses; performance under other modalities, safety policies and deployment pipelines remains to be verified (Limitations section).
- Metric granularity: Automatic metrics such as lexicon ASR, GPT-ASR and toxicity may not fully characterize fine-grained harm severity (Limitations section).
- Representation-level defenses: Joint evaluation with stronger representation-level defenses is left for future work (Limitations section).
- Mechanism level: The authors state that the results are consistent with representation-level vulnerability, but no causal mechanism is established (Conclusion section); the factorization validation supports the expected separation but does not constitute evidence of complete decoupling (Factorization Validation section).
Experimental coverage
- The evaluated models are 3 open-source MLLMs (LLaVA-7B, MiniGPT-4, InstructBLIP) and 3 commercial systems (GPT-4o, Gemini, ERNIE), 6 in total (Section 5.1).
- The benchmarks are AdvBench, MM-SafetyBench, VAJM; TASF training uses 128 requests, and comparison results are reported on 40 representative requests (Section 5.1).
- Defense evaluation includes four: VLGuard, ProEAT, AdaShield, BlueSuffix (Section 5.2).
- Reviewer calibration includes majority voting by 5 human volunteers and two non-GPT reviewers, Qwen3 and DeepSeek (Table 8); the paper does not report the specific sample size of the human review, nor does it report consistency values between human and automatic reviewers.
- The paper does not report the number of repetitions and query budget for each setting; for efficiency it reports about 120s per sample (Figure 2).
Summarize the main content of the paper
TA-SPA optimizes transferable jailbreak perturbations in a text-anchored semantic space, and reports high attack success rates against both open-source and commercial MLLMs under black-box conditions.
A multimodal jailbreak attack paper: the authors argue that MLLM safety alignment is learned in text representation space and is difficult to transfer to fused cross-modal representations, while existing attacks rely on surface visual or lexical cues and have limited transferability.
- Method: TA-SPA uses a frozen CLIP encoder to decompose image-text representations into cross-modal semantic factors and modality residuals (TASF), constructs diverse target anchors by perturbing the text residual while fixing the semantic factor (SPA), then performs PGD optimization in this semantic space with a contrastive margin loss (∥δ∥∞ ≤ 8/255, 100 steps), transferring to victim models under black-box conditions.
- Main results: Among the 9 combinations of LLaVA-7B, MiniGPT-4, InstructBLIP × AdvBench, MM-SafetyBench, VAJM, TA-SPA achieves the highest G-ASR on 8, e.g., LLaVA-7B × AdvBench reaches 82.0% (Table 1).
- Transfer: The average G-ASR against GPT-4o, Gemini, ERNIE is 52.3%, higher than the strongest baseline IDEATOR's 49.4%; on ERNIE, IDEATOR is higher (Table 3).
- Defenses: Under VLGuard, ProEAT, AdaShield, BlueSuffix, the average ASR is highest, ranking first in 10 of 12 combinations (Table 9).
- Control and calibration: The clean visual surrogate itself already has 45.0% G-ASR, and the full method 60.0% (Table 5); human majority-voting ASR is 77.0% (Table 8).
- Authors' conclusion: The results are consistent with representation-level vulnerability, and the authors argue for representation-level safety alignment beyond input-level filtering, but acknowledge that no causal mechanism is established and that validation is only on image-text modality and a fixed set of benchmarks/defenses.