The Scene Is the Attack Surface: Revealing Overlooked Scene-Semantic Safety Failures in Multimodal Large Models

1. 01What problem does this paper attempt to solve?

   The authors claim that scene semantics alone can change refusal behavior and propose SGA and SceneSafetyBench.

   Task-relevant scene semantics alone can shift MLLMs from refusal to compliance, even when the image adds no attack instruction and the original task intent is unchanged.

   • Emergence of scenes: The authors state that MLLMs connect scene evidence composed of entities and relations in the image to the text request, inferring how the activity proceeds and how the request should be understood. If safety mechanisms are unstable to this scene-level semantics, visual grounding becomes a source of safety failure that existing content-based defenses may not detect.
   • Existing approaches: The authors state that prior visual jailbreaks mostly place harmful information in task-relevant objects, which are relatively easy to detect. The Figure 1 caption writes ChatGPT 5.6, while the main text writes GPT-5.6: it refuses the original request, the intent-preserving rewrite, and the image with task-relevant objects.
   • Authors' observation: With the same rewritten request paired with a coherent task scene, the authors state the model gives malicious procedural guidance. In Figure 2, Claude Haiku 4.5 still refuses with isolated objects but gives substantive assistance with a coherent scene that presents roles, relations, and task states. The authors call this a scene-grounded visual safety vulnerability and state these examples do not yet establish the hypothesis.
   • This paper's work: The authors propose Scene-Grounded Attacks (SGA), construct SceneSafetyBench with 400 image-text pairs, and examine visual grounding and decoder intermediate representations on white-box models.
   • Threat model:
     • Target: Mtarget refuses the intent-preserving rewrite q̃ under text-only input, but provides substantive assistance for the same request when paired with scene I. q and q̃ may differ in wording, while the attack intent and expected outcome remain equivalent.
     • Knowledge: The adversary is black-box, with access only to Mtarget's standard image-text interface. White-box access is used only for post-hoc diagnostics, not for constructing or executing the attack.
     • Capability: Can pair q̃ with scene evidence I for the same task. The scene encodes no independent instruction, and no optimization is done using Mtarget's parameters, gradients, or responses. Each image-text pair is evaluated independently, and prior queries or responses cannot be read.
     • Victim system: The victim in Section 3.1 is the target MLLM Mtarget; that section does not name a specific model.

   Collapse this answer

2. 02What related research exists?

   The authors distinguish scene organization from visual jailbreaks, contextual attacks, and multimodal safety benchmarks.

   Visual jailbreaks and multimodal safety

   • Modality transfer: The authors state that defenses learned mainly in the language modality do not always transfer to multimodal representations (Hossain and Imteaj, 2025; Xu et al., 2025). The paper only makes this description and does not separately criticize these two works.
   • Harmful semantics relocated: FigStep (Gong et al., 2025) and FC-Attack (Zhang et al., 2025) encode harmful instructions into typographic images or flowcharts.
   • Processing multimodal pipelines: JOOD does distribution shift, ASO learns style triggers, JPS combines visual perturbations with text guidance, JailBound manipulates internal safety boundaries, and VisCRA exploits visual reasoning traces (Jeong et al., 2025; Luo et al., 2026; Chen et al., 2025; Song et al., 2025; Sima et al., 2025).

   Semantic and contextual visual jailbreaks

   • Contextual priming: VisCo constructs fictional image-grounded dialogue histories, MIRAGE uses visual narratives and character immersion to gradually build harmful scenarios, and Furina exploits refusal instability in fragmented multimodal contexts (Ziqi et al., 2025; You et al., 2025; Wu et al., 2026).
   • Task-relevant images: HADES and MM-SafetyBench show that task-relevant images can weaken defenses (Li et al., 2024; Liu et al., 2024). The paper notes they do not isolate the role of scene organization.
   • Contrast with this paper: The paper states that SGA preserves the original task intent, only changes how objects, relations, and states are organized, and independently queries the target model each time, unlike character immersion and dialogue history attacks.

   Multimodal jailbreak benchmarks

   • Attack-oriented: HADES, MM-SafetyBench, JailBreakV-28K, and IDEATOR construct or aggregate jailbreak samples.
   • Unified evaluation: MMJ-Bench, SafeBench, and OmniSafeBench-MM standardize multiple attacks, defenses, risk categories, and evaluation protocols.
   • Context-sensitive: VLSBench, MSSBench, SafeMT, and MIR-SafetyBench respectively examine visual information leakage, situational context, multi-turn interaction, and multi-image relations. The paper notes these benchmarks broaden coverage but do not isolate how the internal organization of a single task-relevant scene affects safety.

   Baseline methods and benchmarks

   • Baselines: QR-Related, FigStep, VisCRA, and four encryption configurations of MML: MML-WR, MML-M, MML-R, MML-B64. The eight risk categories correspond to MM-SafetyBench: Illegal Activity, Hate Speech, Malware Generation, Physical Harm, Economic Harm, Fraud, Pornography, Privacy Violence.
   • Data: SceneSafetyBench; end-to-end comparisons use each method's own attack benchmark. Questions come from MM-SafetyBench and FigStep, supplemented by GPT-5.6 Sol.

   The authors position this paper as studying the relation and task-state semantics of coherent scenes: no visual instructions embedded, no optimized adversarial signals, and scene-level semantics treated as an independent implicit attack surface.

   Collapse this answer

3. 03How does the paper address this problem?

   SGA decomposes tasks, verifies scene coverage, then independently queries and constrains synthesis.

   The authors use controlled scene organization to separate "the image contains task-relevant content" from "grounding the task into a coherent scenario," and then turn this failure into an automatically executable SGA.

   Problem Setting and Failure Definition

   • Notation: q is the original harmful request, q̃ is an intent-preserving rewrite, I is the visual input, and Mtarget is the target MLLM. The intent constraint is written as I(q̃) ≃ I(q), meaning the attack intent and expected outcome remain equivalent, though wording may differ.
   • Failure: Under text-only input, Mtarget refuses q̃, but when paired with I, it provides substantial assistance. The scene provides contextual evidence for the same task without encoding independent instructions.
   • Contrasting Intents: The authors aim to separate text rewriting, visual semantic coverage, and scene-level coherence. The example in Section 3.2 is presented as a hypothesis to be tested, not an established conclusion.

   SceneSafetyBench and Five-Level Comparison

   • Instances: Each instance retains the original harmful question and includes an image-text pair. Only the image-text pair is sent to the target model; the original question is used to judge whether the response completes the original harmful goal.
   • Sources and Processing: Questions come from MM-SafetyBench and FigStep, supplemented by GPT-5.6 Sol. The authors identify task-relevant visual elements, retrieve images from the web, and use GPT Image 2 to create Pencil Sketch and Pixel Art, with settings following Adversarial Style Optimization. Low-quality images, poorly aligned image-text pairs, and samples that do not maintain the original task goal are manually removed. The paper does not report the number removed.
   • Scale: 400 pairs, eight categories from MM-SafetyBench. Table 1: Illegal Activity 87, Hate Speech 30, Malware Generation 110, Physical Harm 78, Economic Harm 36, Fraud 28, Pornography 8, Privacy Violence 23.
   • Five-Level Inputs: 50 pairs are drawn from the benchmark for white-box diagnosis. Iscene uses benchmark images; Qwen-Image additionally generates Iitem containing only the main task elements, and Imulti without explicit interactions, causal chains, or task flows. Conditions are (A) original question without image, (B) q̃ without image, (C) q̃+Iitem, (D) q̃+Imulti, (E) q̃+Iscene. q̃ remains unchanged in (B)–(E).

   SGA: Decomposition, Image Generation, and Verification

   • External Models: The fixed proxy Mproxy is Qwen3-VL-8B-Instruct, responsible for analysis, decomposition, subquestions, image generation prompts, alignment refinement, and coverage verification. The text-to-image GT2I is Qwen-Image. The synthesis model Msyn is Qwen2.5-14B-Instruct, used only at the end.
   • Blueprint: The proxy first obtains the intent type, expected output form, and core goal, then splits into required components Breq(q) and optional components Bopt(q). Each semantic path corresponds to a subquestion and a visual evidence description. Subquestions are related to the original request but are not a dialogue sequence.
   • Two Scene Types: Ordinary visual paths use text-to-image. Paths requiring technical reference representations use deterministic templates and do not enter the text-to-image refinement loop. The paper does not report the number of each path type.
   • Alignment: The proxy scores image-text alignment from 1 to 5 with diagnostics. A score of 5 means the image clearly presents the required evidence; 1 means irrelevant or contradictory. The text-to-image path threshold is θalign=4; below threshold, prompts are revised per diagnostics and regenerated, up to Tmax=3 times, keeping the highest-scoring image even if no candidate reaches the threshold.
   • Coverage: Before querying the target model, the proxy checks whether all paths cover Breq(q). If insufficient, up to K=2 additional paths are added, following the same subquestion, image generation, and alignment process. This step does not depend on target model responses.

   Independent Queries, Synthesis, and Judgment

   • Independent Queries: Written as r_i = M_target(q_i, I_i^⋆), meaning each path submits only the current subquestion and its retained image. The target model does not see the original request, other subquestions, or other responses. The authors use this to avoid multi-turn dialogue accumulation.
   • Synthesis: Msyn organizes task-level answers based on the original request, paths, and local responses. The paper states that synthesis must not introduce unsupported content; synthesis is not invoked when there are no target model responses. The provided text does not include the synthesis prompt, so detailed constraints are not supplemented here.
   • Behavior Labels: The main experiment uses GPT-4o-mini, following the H, A, D protocol from OmniSafeBench-MM. The diagnostic experiment uses the same protocol to label responses as refuse or comply. VSFR is the proportion of samples refused under condition (B) that turn to compliance under a given visual condition. Paired binary results for (C)–(E) are tested with McNemar's test.
   • Representation Analysis: Performed only on Qwen3-VL-8B-Instruct and LLaVA-v1.5-13B. TAM examines which visual blocks generated tokens fall into; HiddenDetect-style analysis takes the last input token at each layer, passes through final normalization and the language model head, and compares similarity to predefined refusal and compliance token directions. The paper does not list these direction words in the provided text.

   Collapse this answer

4. 04What experiments did the paper conduct?

   SGA average ASR across nine models is 86.17%; on Gemini, MML-M achieves 98.15%, higher than SGA's 97.94%.

   Experimental setup

   • Target models: white-box are Qwen3-VL-8B-Instruct, GLM-4.1V-9B-Thinking, InternVL3.5-8B-Instruct, MiniCPM-V 4.5, Phi-4-reasoning-vision-15B, Gemma 3 12B IT; black-box are GPT-5.6 Luna, Gemini 3.5 Flash-Lite, Claude Haiku 4.5. Diagnostics additionally use LLaVA-v1.5-13B.
   • Attack-side models: SGA's analysis, decomposition, alignment, and refinement use Qwen3-VL-8B-Instruct, image generation uses Qwen-Image, and synthesis uses Qwen2.5-14B-Instruct. Thus when Qwen3-VL-8B-Instruct is the target, the proxy and target are the same model.
   • Shared inference: target queries use temperature 0.7, seed 42, and a maximum of 1024 tokens. Both evaluation suites use this setting and the same review protocol.
   • Two task suites: In the end-to-end evaluation of Section 7, each original harmful question is one instance, Ri is the synthesized task-level response, and partial responses are not scored. Section 8 sends each image-text pair of SceneSafetyBench directly to the target model, then scores against the original harmful question. The two should not be read together.
   • Baselines and scale: The seven configurations are QR-Related, FigStep, VisCRA, MML-WR, MML-M, MML-R, MML-B64, and the eight categories correspond to MM-SafetyBench. The paper states the number of samples differs across methods but does not give the counts. SceneSafetyBench has 400 pairs, with per-category counts in Table 1, each pair three times.
   • Metrics and review: H is 1–10, A and D are 1–5, averaged over three trials and all instances. The reviewer is GPT-4o-mini. A violation requires H≥3 and A≥3; a trial succeeds when S≥3, i.e., additionally requiring D≥3. If any of the three trials succeeds, the instance counts toward ASR. VisCRA only uses Qwen2.5-VL to select occlusion regions; it is not a target model.

   Main results

   The table below is Table 4's ASR. Columns keep only SGA, the baseline with the highest average ASR (MML-M), and FigStep. Only eight rows can be shown for the nine models; InternVL3.5-8B-Instruct is omitted.

   Swipe horizontally to read the table

   Target model	SGA	MML-M	FigStep
   MiniCPM-V 4.5	94.75%	93.42%	71.60%
   Phi-4-reasoning-vision-15B	91.26%	89.61%	4.94%
   Qwen3-VL-8B-Instruct (proxy equals target)	90.12%	88.17%	28.81%
   GLM-4.1V-9B-Thinking	89.09%	87.04%	45.42%
   Gemma 3 12B IT	88.27%	80.45%	39.92%
   GPT-5.6 Luna	70.16%	63.89%	21.30%
   Claude Haiku 4.5	65.33%	11.73%	33.33%
   Gemini 3.5 Flash-Lite	97.94%	98.15%	37.65%

   • Ranking: The authors state SGA's ASR is highest on eight models. In the table, Gemini 3.5 Flash-Lite is the exception. The authors state the advantage over each model's strongest baseline ranges from 1.33 to 2.57 percentage points across seven models; on Claude Haiku 4.5 the strongest baseline is FigStep. On the omitted InternVL3.5-8B-Instruct, SGA ASR is 88.58% (Table 4).
   • Fine-grained metrics: The authors state SGA's H is highest on all nine models, A is highest on six and tied on another, and D is highest on eight. The nine-model arithmetic means are H 7.69, A 4.70, D 4.54, ASR 86.17%, higher than the model averages of the strongest baseline per metric by 0.58, 0.31, 0.30, and 8.55 percentage points.
   • Remaining baselines: The authors state the model-average ASR for QR-Related, VisCRA, and FigStep is 21.75%–35.86%, and for the four MML variants 61.23%–77.62%, still on average below SGA. SGA ASR exceeds 88% on all six white-box models, from Gemma 3 12B IT's 88.27% to MiniCPM-V 4.5's 94.75%.

   SceneSafetyBench single-pair results

   • What was measured: Nine target models directly answer 400 image-text pairs, each three times, scored against the original harmful question (Table 3). This is not the SGA pipeline score.
   • Results: Model-average ASR is 64.33%, with H/A/D of 5.23/4.35/3.99. Overall ASR ranges from Phi-4-reasoning-vision-15B's 15.00% to GLM-4.1V-9B-Thinking's 93.75%. Gemma 3 12B IT and MiniCPM-V 4.5 also exceed 91%; Qwen3-VL-8B-Instruct and InternVL3.5-8B-Instruct both exceed 81%. GPT-5.6 Luna is 34.50% and Claude Haiku 4.5 is 31.00%.
   • Authors' interpretation: The authors state this kind of failure is common, but model differences are large, and ASR can mask response nature. Gemini 3.5 Flash-Lite has A=4.90 and D=4.68 yet an ASR of 58.50%. The model-average ASR for Malware Generation is 82.83% with H 6.55; the authors state eight models have the highest ASR in this category (including ties). Except for Phi-4, models range from 75.45% to 100% in this category, and the three black-box models range from 75.45% to 94.55%. The model-average ASR for the remaining seven categories is 50.00%–60.15%; among them, Pornography has only 8 pairs (Table 1).

   White-box diagnostics

   • Qwen3-VL: 50 pairs, five-level conditions (Table 2). Text-only ASR is 4.0% and 0.0%. item, multi-item, and scene have ASR and VSFR both at 40.0%, 66.0%, and 76.0%, with scene's 95% CI at [62.6, 85.7]. McNemar p for item vs. multi-item and item vs. scene is 0.0044 and 2.8×10^-4; multi-item vs. scene is 0.267. The paper marks p<0.05 as significant. The authors state the degradation cannot be explained by isolated objects alone, and the difference between multi-object and scene does not reach that threshold.
   • LLaVA-v1.5-13B: Text-only ASR is already 50.0% and 44.0%. On the 28 cases that still refuse, the three-level VSFR is 78.6%, 75.0%, and 82.1%, with scene's CI at [64.4, 92.1]; the three pairwise p values are 1.000, 1.000, and 0.500. The authors believe this model already readily complies with text and isolated visual inputs, so additional scene structure is hard to separate further. The paper does not state whether these 50 pairs were repeated three times.
   • Grounding and layer-wise representations: The authors state that in Figure 3, TAM highlights task-relevant regions and varies with isolated objects, multiple objects, and scenes; the caption notes the upper example is LLaVA-v1.5-13B and the lower is Qwen3-VL-8B-Instruct. The authors state that neither spatial concentration nor redistribution reliably predicts safety flips, nor does it isolate the contribution of relational structure. The difference between scene-flip and stable-safe, the authors state, appears mainly in the decoder's middle-to-last third, manifesting as refusal shifting toward declining safety margin. Layer-wise tests use Mann–Whitney U with Benjamini–Hochberg FDR; nominal layer-wise p values are exploratory only; the paper does not report these p values. LLaVA's stable-safe group is smaller, which the authors state limits statistical power.

   Defenses and review consistency

   • Text side: Same 50-question subset as Section 7.3. On Qwen3-VL, SGA without defenses has an ASR of 100%; in that row the proxy and target are the same model. Llama Guard 3 flags only 3 subproblem inputs among the 50 attack instances; after removing the corresponding paths, ASR drops from 100% to 98%. NeMo Guardrails flags 0 under the default prefix–suffix perplexity threshold. The paper does not give the value of that threshold.
   • Image side: ShieldGemma blocks 148/398 generated scenes, an image-level blocking rate of 37.2%. The authors state 62.8% still pass, and that independently filtering text or vision cannot fully handle this failure. Post-blocking end-to-end ASR is reported only for Qwen3-VL.
   • Human comparison: On a random 50 responses from InternVL3.5-8B-Instruct, three annotators independently score using the same H/A/D scale, with success requiring H≥3, A≥3, and D≥3; at least two successes count as human success (Table 6). Human-aggregated ASR is 78.00%, GPT-4o-mini is 72.00%, binary agreement is 94.00%, at the upper end of the 86.00%–94.00% range among humans. The reviewer's H/A/D means are 6.42/4.10/3.62, and the human aggregate is 5.47/4.03/3.88. The authors state this protocol supports binary jailbreak judgments, while exact scores vary by evaluator.

   Other ablations and analyses

   • Table 5, fixed 50 questions, four models: Full SGA average ASR 95.00%, with scenes removed 30.50% (the paper gives a drop of 64.50 percentage points), with coverage supplementation removed 89.50%, and without target evidence 8.00%.
   • Per-model Full: Qwen 100.00%, GLM 98.00%, InternVL 98.00%, Claude 84.00%; with scenes removed: 42.00%, 28.00%, 34.00%, 18.00%. Qwen's proxy and target are the same model.
   • With coverage supplementation removed: 94.00%, 90.00%, 92.00%, 82.00%. Without target evidence, all four models are 8.00%. The authors state the synthesis stage alone contributes little; the pipeline mainly relies on target-model information elicited by scene-grounded queries.
   • TAM entropy under the item condition: Qwen3-VL flip vs. stable is 0.859 vs. 0.860; LLaVA is 0.885 vs. 0.866, p=0.088.
   • JSD for scene transition (C)→(E): Qwen3-VL is 0.053 vs. 0.088 (p=0.883); LLaVA is 0.070 vs. 0.068 (p=0.908).
   • Exact-score agreement among humans: H is 32.00%–46.00%, A is 36.00%–68.00%, D is 52.00%–62.00%; exact H/A/D agreement between reviewer and humans is 40.00%–52.00%, 42.00%–86.00%, 52.00%–78.00% (Table 6).

   Collapse this answer

5. 05What are some points that can be further explored?

   The paper has no separate limitations section; the conclusion proposes a need for safety alignment that resists coherent scenes.

   Limitations and future directions identified by the authors

   The paper has no separate Limitations or Future Work section. There are two explicitly written future directions:

   • Stable safety decisions: The Conclusion states that the results of existing input-side defenses highlight a need for multimodal safety alignment to remain stable under semantically coherent visual contexts.
   • Joint semantics: Section 9 Discussion states that independently filtering text or visual inputs cannot fully handle scene-grounded safety failures, and points toward safety mechanisms that consider the joint semantics of the request and its visual context.

   Section 4.1 states that LLaVA's stable-safe group is smaller, which limits statistical power. That sentence is not in Limitations, Discussion, Conclusion, or Future Work, so it is not listed in this paragraph. The ethics statement describes controlled experiments and release practices, and is also not listed.

   Experimental coverage

   • Target models: End-to-end and SceneSafetyBench use nine MLLMs, six white-box and three black-box; temperature 0.7, seed 42, maximum 1024 tokens, each instance three times (Section 6, Table 3, Table 4).
   • Data and baselines: SceneSafetyBench has 400 pairs across eight categories, with counts in Table 1. End-to-end baselines are seven configurations; source requests come from MM-SafetyBench, and the paper states the number of samples differs across methods but does not report these counts (Sections 6.2, 7).
   • Diagnostics: White-box analysis uses 50 pairs, Qwen3-VL-8B-Instruct and LLaVA-v1.5-13B, five-level conditions, and reports McNemar, TAM, and hidden-state analyses. The paper does not state the number of repetitions for these 50 pairs, nor does it give layer-wise p values (Section 4.1, Table 2).
   • Ablations and defenses: Ablations and defenses use the 50-question subset from Section 7.3; ablations have four target models (Table 5). Defenses are Llama Guard 3, NeMo Guardrails, and ShieldGemma, with post-blocking end-to-end ASR reported only for Qwen3-VL (Section 9). The paper does not report SGA's actual query counts or refinement round distribution; the budget stated in the method is Tmax=3, K=2.
   • Review: Automatic review uses GPT-4o-mini. Human comparison uses 50 InternVL3.5-8B-Instruct responses and three annotators (Section 8.1, Table 6). The proxy, image generation, and synthesis models are Qwen3-VL-8B-Instruct, Qwen-Image, and Qwen2.5-14B-Instruct, respectively.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors claim that scene semantics can change safety decisions while intent remains unchanged, with SGA achieving an average ASR of 86.17% across nine models.

   The authors treat relationships and task-state semantics in coherent scenes as an attack surface for MLLMs and provide an automatically constructed scene-grounded attack.

   • Problem: When images do not write new attack instructions and the original task intent remains equivalent, can refusal turn into substantial assistance by only changing how task evidence is organized into scenes? The authors argue this is not the same as "whether task-related objects appear in the image."
   • Approach: A proxy model extracts a semantic blueprint, decomposes the request into complementary subquestions, generates scenes, then performs 1–5 alignment and coverage verification; the target model sees only one image-text pair at a time; another model synthesizes local responses into a task-level answer. A new SceneSafetyBench with 400 pairs and eight categories is built. Constructing the attack does not require white-box access to the target model.
   • Pipeline Results: Across nine target models, the average ASR of synthesized responses is 86.17%, with H/A/D of 7.69/4.70/4.54, and ASR is 8.55 percentage points higher than the strongest baseline. SGA is highest on eight models; on Gemini 3.5 Flash-Lite, MML-M achieves 98.15% and SGA 97.94%. Among them, Qwen3-VL-8B-Instruct serves as both proxy and target, with an SGA ASR of 90.12%.
   • Single-Pair and Diagnostics: When directly answering benchmark image-text pairs, the nine-model average ASR is 64.33%, with the low end being Phi-4-reasoning-vision-15B at 15.00%. In 50 controlled inputs, Qwen3-VL's scene VSFR is 76.0%, with McNemar p for item-to-scene at 2.8×10^-4; for LLaVA, p≥0.5 across the three visual conditions. Removing scenes drops the four-model average ASR on 50 questions from 95.00% to 30.50%.
   • Author Conclusions: The authors believe spatial grounding alone cannot explain the safety flip; the difference appears more after visual evidence enters mid-to-late safety representations. ShieldGemma blocks 148/398 generated scenes, i.e., 37.2%. The authors argue that safety decisions should also remain stable under semantically coherent scene-grounded changes.

   Collapse this answer

Related papers