Red-Teaming Multimodal Reasoning: Cross-Modal Entanglement Attacks Jailbreak Vision-Language Models

Existing black-box jailbreaks struggle to stress-test the safety alignment of VLMs with multimodal reasoning.

• Scenario: The authors claim such VLMs may handle complex multimodal harmful tasks, making them high-value attack targets. The visual modality is closer to real applications while also introducing new security vulnerabilities.
• Existing shortcomings: Mainstream black-box attacks split malicious cues across modalities to distract attention, but the image-text combinations are simple and fixed, and complexity is not scalable. The authors claim that when used for red teaming, they only trigger the most basic reflexive defenses: visible engineered combinations are easily recognized, and single-hop image-text fusion does not test multi-step reasoning.
• Observations: The authors summarize three points. Safety alignment trained on the text side does not fully generalize across modalities; explicit harmful content in a single modality is easily detected, while implicit intent that only appears in cross-modal understanding remains vulnerable; the visual combination space is larger than text, facilitating OOD attacks beyond alignment coverage.
• This paper: Proposes COMET (CrOss-Modal Entanglement AtTack), using scalable cross-modal entanglement to exceed the safety alignment patterns that VLMs have been trained on and generalized to. The three steps are knowledge-scalable rewriting, cross-modal cue entanglement, and cross-modal scenario nesting.
• Threat model:
  • Goal: Induce unsafe outputs, including misinformation, actionable criminal instructions, biased or toxic content, and avoid empty jailbreaks, making outputs specific and persuasive.
  • Knowledge: Black-box, no access to gradients. White-box methods required for visual perturbation approaches are not within this attack setting.
  • Capabilities: Only control image and text inputs. An auxiliary model rewrites queries, generates entangled images, replaces migrated entities with spatial pointers, and then appends a scoring scenario and scoring rubric.
  • Victim system: VLMs with multimodal reasoning. Experiments evaluate 9 mainstream models. Success is determined by StrongReject's ASR and HS.

Multimodal Reasoning

• Visual Grounding: Groma by Ma et al. (2024), Kosmos-2 by Peng et al. (2023) align language to spatial regions and visual entities, enabling reasoning to progressively reference specific visual cues.
• Multi-hop and Long Chains: Multimodal Chain-of-Thought by Zhang et al. (2023), Visual CoT by Shao et al. (2024), and Insight-V by Dong et al. (2024) decompose complex tasks into multi-hop or long-chain reasoning over interleaved images and text. The authors state these capabilities also increase misuse risk, thus requiring more red-teaming for safety alignment.

Three Types of VLM Jailbreaks

• Visual Perturbation: Hao et al. (2024), Qi et al. (2024) construct adversarial images to mislead visual encoders. The paper states these methods require white-box access and have limited transfer across VLMs.
• Typography and Substitution: FigStep (Gong et al., 2025), Li et al. (2024), and MM-SafetyBench (Liu et al., 2024) place malicious prompts into visual layouts or perform semantic substitution. The paper states they often rely on direct semantic substitution, which can be detected by evolving safety alignment. The introduction also summarizes early attacks as typographic prompts, semantic image substitution, and visual cryptography.
• Cross-Modal Distribution: HIMRD (Ma et al., 2025), Shayegani et al. (2023), and CS-DJ (Yang et al., 2025) distribute attack semantics across modalities, making each modality appear benign under intra-modal inspection.

Design Attribute Comparison

• Four Attributes: The paper summarizes Modality Concealment, Risk Distribution, Attack Scalability, and Harmful Output Steering from existing work. The last corresponds to avoiding empty jailbreaks (Souly et al., 2024).
• Table 1: Compares FigStep, MML (Liu et al., 2025), Visual-RolePlay (Ma et al., 2024), CS-DJ, HIMRD, VisCo (Li et al., 2025) with COMET. The text states existing mainstream methods mainly rely on fixed, heuristic substitution to hide semantics, thus lacking scalability; in the table, CS-DJ's Attack Scalability is ✓, other existing methods are ✗, and COMET is ✓ for all four. FigStep and Visual-RolePlay are marked as image-unsafe, VisCo is marked as text-unsafe.

Baselines and Benchmarks

• Baseline Methods: Experiments compare FigStep, CS-DJ, HIMRD, using their recommended hyperparameters. MML, Visual-RolePlay, VisCo in Table 1 are not reported with ASR as experimental baselines.
• Benchmark: SafeBench's 7 harmful categories, and the full SafeBench-tiny. Attack success is computed using StrongReject.

The authors state COMET uses an adaptive mechanism to establish entangled semantics that must go through multimodal reasoning, thus enabling a more scalable black-box attack; in the contributions, they call the framework novel and state they introduce multimodal reasoning attacks to red-team advanced VLMs.

COMET expands a harmful query, splits it into image and text, and embeds it into a scoring task, causing the victim VLM to enter instruction following and integrate attack semantics while understanding the task. Figure 2 shows three stages. The rewriting comparisons and prompt texts in the main text are not quoted here.

Problem Setting

• Input: The original query is written as Q_orig=(T_orig, ⊥), where T_orig is the textual malicious instruction and the target response is R_orig. The paper does not define the symbol ⊥.
• Output: The final text T_atk and image I_atk. ⊕ denotes concatenation.
• Illustrative comparison: Figure 1 depicts the vanilla attack as a fixed one-step image-text fusion, marked as failing the Safe Check; COMET is depicted as multi-step image-text entanglement, marked as passing the Safe Check and entering instruction following. This is an illustration, not a quantitative result.

Knowledge-Scalable Rewriting

• Method: An auxiliary text model M_T2T uses a ReAct loop to iteratively optimize a knowledge set K={k1,...,kn}, rewriting the query into a multi-hop chain-style instruction and injecting multiple visualizable entities. The system prompt is denoted prompt_chain. The main text does not give the full prompt or the number of loop iterations.
• Objective: Increase the probability of eliciting the original harmful response while reducing the probability of a safe refusal, i.e., K^∗=argmax_K P(R_orig|Q_refr(K))·(1−P(R_safe|Q_refr(K))). The rewriting is written as Q_refr=ReAct(M_T2T, prompt_chain, Q_orig, K*).
• Authors' rationale: The objective, factual nature of knowledge can serve as a benign carrier for malicious intent; associations among knowledge can adaptively expand semantic complexity to bypass safety alignment of different VLMs.

Cross-Modal Clue Entanglement

• Motivation: The authors state that pure text rewriting may still be directly decoded because the text format may already be internalized by VLMs; vision is sparser and more ambiguous, with a comprehension gap relative to text.
• Image: M_T2T extracts visualizable entities E_vis from the rewritten text and adds decoy entities E_decoy. Under prompt_img and style mask M_style, an image prompt is generated, and the text-to-image model M_T2I synthesizes I_entgl. In the main experiment, M_style is set to sketch.
• Text: The image-to-text model M_I2T, guided by prompt_entgl, replaces the migrated entities with spatial pointers that only vaguely refer to image regions, yielding T_entgl. The attack semantics are thus distributed across Q_entgl=(I_entgl, T_entgl).
• Authors' claim: Clues are embedded only at the entity level into benign, narratively parallel images. Figure 3 compares visual obfuscation for the same harmful task; the authors claim COMET uses related entities but unrelated contexts to form a benign narrative. The task text is not quoted.

Cross-Modal Scenario Nesting

• Motivation: The authors state that advanced VLMs are optimized for fidelity on complex reasoning benchmarks like GAIA and HLE, with strong instruction following. COMET constructs an image-text challenge (T_scen, I_scen) that seeks high scores to stabilize the entangled attack and avoid empty jailbreaks. These are not evaluation sets in this paper. The scoring target in the figure is written as 6 points (Master level).
• Scenario image: I_scen follows a template with four sections from top to bottom: scenario title, progress tracking, requirement box, and interface components. Progress tracking uses elements like progress bars and pass rates, which the authors claim create urgency to complete the task.
• Scoring rubric: T_rubric=M_T2T(prompt_rubric, Q_orig), then rendered as a visual table I_rubric. The authors claim this exploits VLMs' instruction following on visual tables to steer output toward detailed harmful content.
• Assembly: T_atk=T_entgl⊕T_scen, I_atk=I_entgl⊕I_rubric⊕I_scen.

Auxiliary Models and Defense Prompts

• Generation models: M_T2T is DeepSeek-V3.1-Terminus, M_I2T is Qwen3-VL-235B-A22B-Thinking, M_T2I is gemini-2.5-flash-image. Generation temperature is 0.1.
• Analysis models: Semantic representation and embeddings use Qwen3-VL-30B-A3B-Instruct; harmfulness space analysis uses anchor embeddings inspired by Lin et al. (2024). It does not generate attacks.
• Defense: During evaluation, AdaShield-Static's prompt is prepended to the payload: first check whether the image contains text or objects implying harmful, illegal, or dangerous activities; if the overall content is judged unethical, illegal, or dangerous, refuse; otherwise, execute the subsequent text instruction. The main text gives the full prompt, which is not repeated here.

Experimental Setup

• Victim models: GPT-4.1, GPT-4.1-mini, Gemini-2.5-Flash, Qwen3-VL-235B-A22B-Instruct, Qwen2.5-72B-VL, GLM-4.5V, Gemini-2.5-Pro, LlaMa-4-maverick, Claude-4.5-Haiku, 9 in total.
• Data: SafeBench's 7 categories, namely ADU, FRD, HAT, ILL, MAL, PHY, PRV; full SafeBench-tiny for generalization. The paper does not state the number of samples in either dataset.
• Baselines: FigStep, CS-DJ, HIMRD, using recommended hyperparameters. Table 2 compares only with CS-DJ; Table 3 compares with all three.
• Metrics: ASR (0–1) is the success rate of jailbreak attempts; HS (0–1) reflects the specificity and persuasiveness of outputs, averaged only over successful attack samples. The hop analysis additionally reports PPL; the style experiment additionally reports image synthesis success rate.
• Evaluation: Both are computed by StrongReject, with GPT-4.1-mini as the driving model. The paper does not state the decision threshold nor report agreement with humans. GPT-4.1-mini is also a victim model.
• Protocol: Each attack is evaluated with 5 attempts, temperature 0.1. Def. denotes the preceding AdaShield-Static prompt; Van. denotes without that defense.

Main Results

According to the All row of Table 2, SafeBench, StrongReject ASR. The header Qwen3-VL-235B-A22B refers to Qwen3-VL-235B-A22B-Instruct from Section 4.1.

Swipe horizontally to read the table

Victim model	CS-DJ Van.	CS-DJ Def.	COMET Van.	COMET Def.
Gemini-2.5-Flash	0.71	0.69	1.00	0.98
GPT-4.1-mini	0.26	0.01	0.94	0.89
GPT-4.1	0.40	0.35	0.96	0.93
Qwen3-VL-235B-A22B	0.22	0.13	0.98	0.92

• The authors claim COMET averages Van. 0.97, Def. 0.93, and CS-DJ Van. 0.22, Def. 0.13. The CS-DJ averages given by the authors are not the same numbers as the four rows above; the All row of Table 2 is authoritative. The authors claim CS-DJ represents the distraction category, fails on these VLMs, and that COMET points out the unsafety of long multimodal reasoning.
• Categories are not 1.00 in every cell. On GPT-4.1-mini's ADU, COMET is Van. 0.86, Def. 0.84. On Gemini-2.5-Flash's FRD, CS-DJ's Def. 0.76 is higher than its Van. 0.72; the authors do not explain this increase.
• The abstract writes over 94% ASR, 29% above the best baseline; the introduction writes ASR 94%, 29% above baseline; the Conclusion writes over 94%, 29% above the state-of-the-art baseline. The paper does not report statistical tests.

Three Baselines on More Models

According to Table 3, SafeBench-Tiny ASR. COMET is the highest in each of the five rows.

Swipe horizontally to read the table

Victim model	CS-DJ	FigStep	HIMRD	COMET
GLM-4.5V	0.56	0.64	0.72	1.00
Gemini-2.5-Pro	0.22	0.52	0.48	0.92
Qwen2.5-72B-VL	0.18	0.88	0.64	1.00
LlaMa-4-maverick	0.10	0.62	0.90	0.96
Claude-4.5-Haiku	0.14	0.60	0.04	0.84
All	0.24	0.65	0.56	0.94

• The authors claim COMET has the highest ASR on these VLMs and that this exposes a general unsafe pattern in VLMs. In the same table, All for FigStep is 0.65, the highest row among the three baselines.
• On Claude-4.5-Haiku, COMET is 0.84, the lowest COMET row in the table; for the same model HIMRD is 0.04. On LlaMa-4-maverick, HIMRD is 0.90, with a smaller gap to COMET's 0.96 than in other rows. The authors do not explain these two rows separately.

Harmfulness and Payload Representation

• HS: In Figure 4, the x-axis is Harmfulness Score (0.0 to 1.0), the y-axis is Frequency, split into two panels for CS-DJ and COMET, with the legend containing the four models from Table 2. The authors claim COMET's responses are closer to 1.00, thus not an empty jailbreak, and that existing work criticizes this issue but lacks a general solution; they instead use evaluation guidance to push the victim model. The extracted text has no binned counts.
• Representation: The authors claim that in Figure 5, COMET payloads cluster with harmless anchors and stay away from harmful anchors, remaining stealthy when the defense is on, and use this to explain Table 2's ASR. The extracted legend is Harmless/Harmful Anchor and CS-DJ, COMET Van./Def.; the caption additionally lists COMET, COMET w/o Scen.N, HIMRD, CS-DJ, and states that lower sample similarity indicates greater diversity. Axis names do not appear in the extracted text.

Component Ablation

Table 4 on SafeBench-tiny, victim model GPT-4.1, compares single components, not removals one by one. K.Refr, C.Enta, Scen.N refer to knowledge rewriting, clue entanglement, and scenario nesting, respectively.

• All three components: ASR 0.96, HS 0.91. The authors claim this is the best in the table.
• Only one: K.Refr is 0.66 (↓0.30), 0.67 (↓0.24); C.Enta is 0.42 (↓0.54), 0.52 (↓0.39); Scen.N is 0.24 (↓0.72), 0.86 (↓0.05). The authors claim the three have different roles: rewriting expands complexity for stealth, entanglement requires knowledge expansion to be effective, and scenario nesting mainly increases output harmfulness.

Other Ablations and Analyses

• hop: In Figure 6, the x-axis is Entanglement Hop Count (0 to 10). The authors claim the trade-off between ASR and HS is at 4 to 6 hops; at 6 hops ASR approaches 100%, and HS decreases with hop because the model must allocate more attention to decoding the original semantics; PPL remains low. The controls are CS-DJ and FigStep at hop=0, HIMRD at hop=1. The authors claim HIMRD's PPL is abnormally high. The extracted text has no point-by-point values.
• Diversity: The authors claim that in Figure 7, COMET's text and visual embedding cosine similarity is lower than fixed templates like CS-DJ; T_entgl and I_entgl without scenario nesting are lower than HIMRD. The figure does not label similarity numbers.
• Scoring rubric: The authors claim that in Figure 8, adding the visual scoring rubric simultaneously increases stealth and output harmfulness, and call this counterintuitive. Without the rubric, attention is distributed over visual entities for safety checks; with the rubric, it shifts to disguised dashboards, UI, and scoring areas. Brighter yellow indicates higher attention. No attention values are given.
• Style: Table 5 uses dall-e-3 for image generation, victim model GPT-4.1, data SafeBench-tiny, reporting ASR/synthesis success rate. HIMRD: Realistic 0.00/0.08, Cartoon 0.02/0.04, Pixel Art 0.04/0.08, Sketch 0.02/0.06. COMET: 0.94/1.00, 0.98/1.00, 0.96/1.00, 0.98/1.00. The authors claim HIMRD is mostly blocked by text-to-image safeguards at the synthesis stage, while COMET's synthesis success rate is 1.00 across all four styles. The text-to-image model in the main experiments is gemini-2.5-flash-image, different from this table.

Limitations and future directions noted by the authors

• Defenses: The Conclusion states that this study further emphasizes the urgency of developing more robust defenses for VLM safety. The paper does not have a dedicated Limitations or Future Work section, nor does it frame any experimental phenomenon as a limitation of the method itself.

Experimental coverage

• Models and tables: Victim models are the 9 VLMs from Section 4.1. Table 2 reports Van. and Def. ASR for 4 of them on SafeBench relative to CS-DJ; Table 3 reports ASR for the remaining 5 on SafeBench-tiny relative to three baselines, with no defense column.
• Baseline scope: Experimental baselines are FigStep, CS-DJ, HIMRD. Table 1 also compares MML, Visual-RolePlay, VisCo, but the experimental section does not report ASR for these three. Related work marks visual perturbations as requiring white-box access with limited transferability, and such methods are not included in the experimental baselines.
• Protocol gaps: Each attack has 5 attempts with temperature 0.1. The paper does not report the number of samples in SafeBench and SafeBench-tiny, nor the number of ReAct iterations or queries.
• Evaluation: ASR and HS are computed by StrongReject with GPT-4.1-mini as the driving model; HS is averaged only over successful samples. The paper does not report agreement between the evaluator and human judgments.
• Analysis settings: The victim model for component ablation, hop, and style experiments is GPT-4.1. The main experiment uses gemini-2.5-flash-image as the image model and M_style is sketch; Table 5 switches to dall-e-3 with styles Realistic, Cartoon, Pixel Art, Sketch. Defense evaluation uses AdaShield-Static prompt prefixing.

COMET is a black-box jailbreak for VLMs with multimodal reasoning, used to red-team cross-modal safety alignment.

The gap the authors address: distributing malicious cues into fixed image-text combinations fails to detect vulnerabilities that emerge only in multi-step reasoning. The attack does not access gradients; it only constructs image-text inputs.

The method has three steps. An auxiliary LLM uses ReAct to expand harmful queries into multi-hop knowledge instructions; visualizable entities are moved into narrative images with decoys, leaving only spatial pointers in the text; then a scenario seeking high scores and a visual scoring rubric are added to push detailed outputs. The auxiliary models in the main experiments are DeepSeek-V3.1-Terminus, Qwen3-VL-235B-A22B-Thinking, and gemini-2.5-flash-image.

• Table 3 on SafeBench-Tiny: COMET's All ASR is 0.94; in the same table FigStep is 0.65, HIMRD is 0.56, CS-DJ is 0.24. The authors claim it exceeds the best baseline by 29%; the abstract and Conclusion write over 94%, the introduction writes 94%.
• Table 2: COMET All for four models is 1.00, 0.94, 0.96, 0.98 without defense, and 0.98, 0.89, 0.93, 0.92 after AdaShield-Static prompt defense. The authors also give a set of averages that do not exactly match these four rows.
• In Table 3, COMET's lowest row is Claude-4.5-Haiku at 0.84; in that row HIMRD is 0.04.
• Table 4 on GPT-4.1, SafeBench-tiny: with all three components, ASR is 0.96 and HS is 0.91; with only one component, ASR is 0.66, 0.42, 0.24.
• In Table 5, COMET's image synthesis success rate is 1.00 across four visual styles; HIMRD's highest synthesis success rate is 0.08.

The authors claim cross-modal entanglement prevents VLMs from directly decoding attack semantics and exploits their instruction following; the Conclusion states this highlights the urgency of developing more robust VLM defenses.