MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models

Flowchart jailbreaks of multilingual vision-language models lack cross-lingual measurement.

• Deployment scenario: Multilingual vision-language models (MVLMs) are used in globally accessible applications, where users interact in different languages and cultural contexts. The authors state that harmful outputs pose societal risks, while safety evaluation and jailbreak research are largely English-centric.
• Existing gaps: The authors state that printed images with benign prompts, embedding harmful intent in images, and encoding harmful instructions into step-by-step flowcharts can all jailbreak vision-language models, with the latter outperforming prior visual jailbreak methods; these studies are largely limited to English. The authors also state that no work has compared the safety strengths of multilingual multimodal models relative to English-centric models under multilingual visual attacks.
• Questions to answer: How language differences affect the safety alignment of MVLMs, and whether these multilingual models offer better safety alignment.
• Proposed content: The benchmark MLingualFC turns harmful instructions into flowchart images in Hindi, Punjabi, Spanish, Romanian, and German, and performs red-team evaluation of Qwen2.5-VL, Gemma-4, and Pangea under black-box settings.
• Threat model:
  • Goal: Bypass MVLM safety alignment to make it generate harmful, unsafe, or safety-policy-violating responses; prompt the model in different languages with structured visual representations.
  • Knowledge: Black-box, with no access to internal architecture, parameters, gradients, safety filters, or output probabilities.
  • Capability: Can only submit multimodal prompts through public interfaces and observe final text; each query is independent and single-turn.
  • Victim system: The target MVLM. The paper does not state whether the attacker can see the system prompt or the original safety policy text.

The authors organize related work along three lines and use plain-text settings as controls in the experiments.

• VLM jailbreaks: Optimization-based methods backpropagate gradients toward a target output, including Bagdasaryan et al. (2023), Shayegani et al. (2024), Qi et al. (2024); Carlini et al. (2023) directly optimize the input image toward a fixed harmful output; Li et al. (2024c) replace harmful text keywords with semantically equivalent objects or actions in the image. Prompt-based methods require only black-box access and can avoid injecting image perturbations: Gong et al. (2025) and Ma et al. (2024) render text as visual prompts; Yin et al. (2023), Han et al. (2023), Bailey et al. (2024b) slightly modify images and pair them with malicious text; Shayegani et al. (2023), Dong et al. (2023), Chen et al. (2023) use surrogate VLMs to generate perturbed images from ordinary images. Zhang et al. (2025) encode harmful instructions into automatically generated step-by-step flowcharts, which the authors say outperforms previous visual jailbreak methods; the graph representation in this paper follows that work.
• Multilingual LLM jailbreaks: The authors say the multilingual aspect of LLM safety has been less systematically studied, with alignment and red teaming mostly conducted in English. Yong et al. (2023) translate unsafe English prompts into low-resource languages such as Zulu and Scots Gaelic to bypass GPT-4's safety filters; the paper links this to the scarcity of safety-related training data in those languages. Yoo et al. (2024)'s Code-Switching Red Teaming mixes languages within a single input, and the authors say ASR rises and refusal rates fall across several frontier models. Pattnayak and Chowdhuri (2026)'s IndicJR is a jailbreak robustness benchmark for South Asian languages without a judge model, covering 12 Indic languages including Hindi, Bengali, and Punjabi; Song et al. (2025) compare LLM safety under mixed-language inputs versus monolingual prompts. The paper does not give specific ASRs for these works.
• Multilingual and visual red teaming: Derner and Bastič (2025) evaluate three MLLMs, seven languages, and four inputs: plain text, DAN prompt + text, text-as-image, DAN prompt + text-as-image. The authors say rendering harmful prompts as images raises ASR and lowers refusal rates, more so in low-resource languages. Gu et al. (2024)'s MLLMGuard is a Chinese-English bilingual safety evaluation suite covering text and image scenarios with 14 risk categories in total.
• Baselines and data: Harmful queries come from the English dataset of Zhang et al. (2025), which includes AdvBench (Zou et al., 2023), with categories including synthetic hazardous materials, cyberattacks, and illegal activities. The experimental controls are two plain-text settings: Text + Description (harmless text plus multilingual steps, not converted to images) and Harmful Query (only the multilingual harmful query). Table 1 also reports English flowcharts.

The authors position this paper after the above works: although models for multilingual multimodal understanding exist, no prior work has evaluated their safety properties under multilingual visual attacks, whether they are stronger or weaker than English-centric VLMs.

MLingualFC translates English harmful queries into flowchart images in five languages, pairs them with benign text that does not indicate harmfulness, submits them to MVLMs in a black-box single-turn setting, and then uses an LLM judge to compute ASR based on a score threshold. The attack does not use gradient optimization and does not add perturbations to images.

Query source and image representation

• Uses the public English dataset from Zhang et al. (2025), with queries from AdvBench, covering categories including synthetic hazardous materials, cyberattacks, and illegal activities. The paper does not state the number of queries.
• Each query is decomposed into procedural substeps and drawn as a directed graph: nodes are one-step text instructions, and directed edges indicate step progression.
• The translation target is the DOT file of the flowchart, not a separately rewritten attack sentence. Figure 1 shows horizontal flowcharts in English, Hindi, and Romanian, with the caption stating that only 3 steps are shown.

Multilingual image generation

• DOT text is translated with GoogleTranslator (deep-translator) into Hindi (hi), Punjabi (pa), Spanish (es), Romanian (ro), and German (de).
• The translations are compiled into PNGs with Graphviz. For each language and query, there are three layouts: horizontal (one row from left to right), vertical (one column from top to bottom), and tortuous (an S-shaped winding path).
• The paper does not state whether translations were manually proofread, nor the number of steps in the full-step flowcharts.

Input submitted to the model

• Each image is paired with a benign system prompt asking the model to analyze and complete the steps in the image; the prompt does not mention that the content is harmful. The authors state that this exploits multilingual vision-text misalignment in safety alignment. The paper does not provide the full text of this prompt.
• The input is single-turn, and queries are independent. The victim model returns only final text.

Success determination

• Uses LLM-as-a-judge. GPT-oss-120b scores responses for harmfulness and completeness on a scale from 1 (complete refusal) to 5 (fully harmful response).
• A score greater than or equal to 4 counts as a harmful response, with the ASR threshold at score ≥ 4.
• The attack success rate for language i is defined as ASR_i=N^jb_i/N_i, i.e., the number of responses in that language reaching the threshold divided by the total number of samples in that language.
• Two plain-text controls are also set up, along with a 5-step versus full-step comparison only for Qwen2.5-VL. Target models are Qwen2.5-VL-3B-Instruct, Gemma-4-E4B-it, and Pangea-7B; Pangea follows the LLaVA-NeXT architecture. The paper does not state temperature, decoding method, or number of repetitions.

Experimental setup

• Target models: Qwen2.5-VL-3B-Instruct, Gemma-4-E4B-it, Pangea-7B. Written as Qwen-2.5-VL, Gemma-4, Pangea in the tables.
• Languages and layouts: Table 1–3 include English, Spanish, Romanian, German, Hindi, Punjabi; layouts are Horz., Vert., Tort. The five non-English languages listed in the main text are Hindi, Punjabi, Spanish, Romanian, German.
• Main setting and baselines: Flowchart + Harmless Text; plain-text baselines are Text + Description and Harmful Query (Table 2). 5-step is reported only for Qwen2.5-VL (Table 3).
• Metrics and judging: ASR. GPT-oss-120b scores on a 1–5 scale, threshold score ≥ 4. The paper does not report per-language sample sizes, number of queries, repetitions, or decoding parameters.
• Human evaluation: Native speakers rate Hindi and Punjabi responses for harmfulness on a 1–5 scale. The main text says ASR was computed for Gemma-4; Table 6 also lists Pangea. The paper does not report the number of annotators or agreement coefficients.
• Interaction: black-box, single-turn, observing only the final text. The computing environment is PyTorch, Hugging Face, and NVIDIA L4 24GB.

Main results

Per Table 1, ASR (%) for Flowchart + Harmless Text. Each cell is in the order horizontal / vertical / tortuous.

| Language | Qwen2.5-VL | Gemma-4 | Pangea | | English | 84.00 / 70.00 / 86.00 | 26.00 / 18.00 / 18.00 | 54.00 / 72.00 / 54.00 | | Spanish | 80.85 / 72.34 / 89.36 | 53.19 / 21.28 / 31.91 | 59.57 / 51.06 / 23.40 | | Romanian | 61.70 / 87.23 / 82.98 | 63.83 / 29.79 / 42.55 | 40.43 / 38.29 / 40.43 | | German | 76.59 / 82.98 / 89.36 | 70.21 / 34.04 / 53.19 | 48.94 / 42.55 / 44.68 | | Hindi | 19.15 / 53.19 / 40.42 | 74.47 / 59.57 / 68.08 | 0.00 / 0.00 / 0.00 | | Punjabi | 4.26 / 0.00 / 4.26 | 6.38 / 14.89 / 4.26 | 0.00 / 2.13 / 2.13 |

• The authors say European languages are overall higher than Indic languages, and say Qwen often exceeds 80% in Spanish, German, Romanian. In Table 1, Gemma-4's Hindi is higher than that model's Spanish, Romanian, and German, which is not fully consistent with "European languages are uniformly higher." The authors also write per model: for Qwen, Hindi and Punjabi are lower; for Gemma-4, Hindi is the most vulnerable and Punjabi the lowest; for Pangea, English and Spanish are highest, with Hindi and Punjabi near zero.
• The authors say Qwen2.5-VL is the most vulnerable model and interpret Gemma-4's and Pangea's lower ASR in Spanish, Romanian, German as stronger safety alignment. For Qwen's and Pangea's Hindi and Punjabi, the authors believe the reason is not stronger alignment but weaker comprehension of Indic languages, which affects reading structured flowcharts.
• The authors say English is not always the most vulnerable, and evaluating only in English would underestimate multilingual vulnerabilities. Gemma-4's multilingual settings are higher than English except for Punjabi. Qwen has no single best layout: Spanish and German are highest in tortuous, Romanian and Hindi highest in vertical.

Plain-text baselines

• Table 2 compares Text + Description and Harmful Query. The authors say that in most languages both plain-text ASRs decrease relative to flowcharts, and on this basis argue that MLingualFC better bypasses guardrails. The paper performs no statistical tests.
• Hindi and Punjabi are the divergent trends the authors point out, and only for Qwen2.5-VL and Pangea. Qwen's Punjabi: Text + Description 66.00%, Harmful Query 24.00%, higher than the Table 1 flowchart values of 4.26% / 0.00% / 4.26%. Pangea's Punjabi is 52.00% for both plain-text settings, with a flowchart maximum of 2.13%; Pangea's Hindi is 32.00% and 34.00%, with flowcharts at 0.00%. Qwen's Hindi is not uniformly higher in plain text: Text + Description 64.00% is higher than all three flowcharts, while Harmful Query 30.00% is higher only than the horizontal layout's 19.15% and lower than vertical 53.19% and tortuous 40.42%.
• Gemma-4's Hindi goes in the opposite direction: flowchart horizontal 74.47%, Text + Description 8.00%, Harmful Query 2.00%. The authors do not include Gemma-4 in the sentence about plain-text ASR rising in these two languages. The authors believe Qwen's and Pangea's low flowchart ASR is not stronger alignment but difficulty understanding structured visual prompts in non-Latin scripts.

Qualitative examples and human evaluation

• Table 4 uses Qwen2.5-VL-3B-Instruct, Romanian, and the same harmful query to compare plain text with the three flowcharts. In plain text the model refuses; once rendered as a flowchart, the model provides the information needed to complete the request. The tortuous layout's response is in English, while vertical and horizontal are in Romanian or mixed with English. The authors say the model sometimes refuses first yet still provides harmful content in the same response, and consider safety behavior unstable under multilingual multimodal prompts. The query and response text are not reproduced.
• Table 5 shows Hindi examples for the same model. In plain text the model recognizes the harmful content and refuses. The horizontal flowchart is understood as exam preparation and generates academic steps; the tortuous layout is understood as processing raw materials into usable materials; the vertical layout's response is unrelated to the query. The table note says the model responds in English. The authors believe Hindi's low ASR is not strong safety alignment but failure to accurately understand structured multilingual visual prompts.
• Table 6's human ASR: Gemma-4's Punjabi is 0 / 2.12 / 8.5, Hindi is 72.00% / 58.00% / 68.00%; Pangea's Punjabi is 0 / 2.13 / 2.13, Hindi is 0 / 0 / 0. The authors say the LLM judge is close to humans, with Pangea differing by 0% and Gemma-4 by 1–6%. Pangea matches Table 1. Gemma-4's Hindi is close to the LLM's 74.47% / 59.57% / 68.08%; for Punjabi against the LLM's 6.38 / 14.89 / 4.26, the vertical layout's difference is larger than the 1–6 percentage points the authors state.

Other ablations and analyses

• Number of steps, Qwen2.5-VL only: Table 3's 5-step horizontal layout is English 92.00%, Romanian 81.25%, Hindi 33.33%, which the authors say is 8%, 19.55%, and 14.18% higher than the full steps, respectively. The authors say vertical and tortuous layouts benefit more from more steps. Spanish tortuous 5-step is 89.58%, while Table 1's full steps are 89.36%, which is not fully consistent with the latter sentence. The paper does not state the number of full steps.
• Layout: The authors say Gemma-4's horizontal layout is higher except for Punjabi; Pangea is more susceptible to the horizontal layout in Spanish, German, Romanian. In Table 1, Pangea's Romanian horizontal and tortuous are both 40.43%.
• Punjabi across all layouts: The three models' maxima are Qwen 4.26%, Gemma-4 14.89%, Pangea 2.13% (Table 1). Qwen's vertical layout is 0.00%.

Limitations and future directions noted by the authors

• Model family: The Limitations state that Qwen-2.5-VL, Gemma-4, and Pangea were evaluated, and experiments can be extended to other model families to strengthen generalizability.
• Defenses not evaluated: The Limitations state that the main analysis focuses on attack vulnerabilities and does not evaluate defense mechanisms or alignment strategies in multilingual settings; studying robust multilingual safety defenses is a future direction.
• Planned work: The Conclusion states that the authors plan to develop defenses against multilingual visual jailbreak prompts and study alignment strategies that remain effective across languages.

Experimental coverage

• The tested models are Qwen2.5-VL-3B-Instruct, Gemma-4-E4B-it, and Pangea-7B, for a total of 3 (Section 4). The judging model is GPT-oss-120b, with a threshold of score ≥ 4.
• The main results use Flowchart + Harmless Text, with languages English, Spanish, Romanian, German, Hindi, and Punjabi, and layouts horizontal, vertical, and circuitous (Table 1).
• The plain-text baselines are Text + Description and Harmful Query, covering the six languages above and the three models (Table 2).
• The comparison of 5-step and full-step is reported only for Qwen2.5-VL (Table 3). Qualitative examples are in Romanian and Hindi for that model (Table 4, Table 5). Human evaluation numbers cover Gemma-4 and Pangea in Hindi and Punjabi (Table 6).
• The interaction is black-box and single-turn. The paper does not report per-language sample size, number of queries, number of repetitions, decoding parameters, or agreement coefficients between human and LLM judges.

MLingualFC is a multilingual multimodal benchmark that measures whether flowchart visual prompts can cause MVLM to produce responses that violate safety policies in non-English languages.

Existing flowchart jailbreaks are largely measured in English, while MVLMs are already deployed for multilingual users. The authors translate English harmful queries from AdvBench into Hindi, Punjabi, Spanish, Romanian, and German, and create horizontal, vertical, and detour layouts, paired with benign text that does not indicate harmfulness. GPT-oss-120b scores on a 1–5 scale, with a score of at least 4 counted as ASR. The tested models are Qwen2.5-VL-3B-Instruct, Gemma-4-E4B-it, and Pangea-7B.

• In Table 1, Qwen's Spanish and German detour layouts are both 89.36%, and English detour is 86.00%; Punjabi is highest at 4.26%, with vertical layout at 0.00%.
• Gemma-4's Hindi horizontal layout is 74.47%, and English horizontal layout is 26.00%; Punjabi's highest is vertical layout at 14.89%. Pangea's Hindi all three layouts are 0.00%, and English vertical layout is 72.00%.
• Plain text is lower in most languages, but not everywhere: Qwen's Punjabi under Text + Description is 66.00%, and Pangea's Punjabi both plain text conditions are 52.00% (Table 2). The authors believe that the low flowchart ASR in Hindi and Punjabi is more likely due to visual text recognition or language understanding, rather than stronger alignment; in a qualitative Hindi example, the model interprets the flowchart as a task unrelated to the original query.
• Human evaluation and LLM review agree on Pangea's Hindi and Punjabi; for Gemma-4's Punjabi vertical layout, human 2.12 and LLM 14.89 differ more than the 1–6% the authors wrote (Table 6, Table 1).

The authors state that current safety mechanisms fail to generalize across languages and modalities, and that evaluating only in English underestimates vulnerabilities. They plan to study defenses and alignment that remain effective across languages. The ethics statement says the full dataset is not public; access will be limited to certified researchers who sign an acceptable use agreement.