MMDT: Decoding the trustworthiness and safety of multimodal foundation models

What MMDT evaluates: where text-to-image (T2I) and image-to-text (I2T) multimodal foundation models are prone to failure in safety, hallucination, fairness, privacy, adversarial robustness, and out-of-distribution (OOD) robustness.

• Context: The authors state that MMFMs have been used in autonomous driving, healthcare, and virtual assistants, while existing work has shown that text-to-image models can generate unsafe content. As deployment scope expands, trustworthiness and safety concerns increase.
• Existing evaluations: The authors state that existing multimodal benchmarks mostly evaluate usefulness, or only cover limited perspectives such as fairness and privacy; existing trustworthiness evaluations mostly target specific perspectives (Qiu et al., 2022; Cho et al., 2023).
• Authors' proposal: The authors propose MMDT (Multimodal DecodingTrust) and state it is the first unified MMFM safety and trustworthiness evaluation platform. Each perspective uses data generation pipelines and human verification to construct challenging data.
• Evaluation setup:
  • Systems under test are T2I and I2T models; inputs are text prompts, or images with accompanying questions, respectively.
  • Red-teaming goals are separated by perspective: eliciting harmful outputs, hallucination, group or individual unfairness and over-fairness, training data memorization, sensitive attribute or location inference at inference time, task failure under adversarial inputs, and performance degradation under OOD transformations.
  • Knowledge settings are not uniform. Adversarial robustness performs white-box (targeted) attacks on surrogate models and then transfers; T2I jailbreak adapts SneakyPrompt, targeting DALL·E 2's safety filter; I2T jailbreak optimizes universal adversarial images against a white-box surrogate.
  • Success determination is also separated by perspective: safety uses Bypass Rate (BR) and Harmful content Generation Rate (HGR); the rest use accuracy, fairness scores, or CLIP similarity.

• Capability benchmarks: Goyal et al. (2017), Singh et al. (2021) do single-task evaluation; MMMU by Yue et al. (2024), HRS-Bench by Bakr et al. (2023), and T2I holistic evaluation by Lee et al. (2024) target harder capability tasks. The introduction states such benchmarks mainly evaluate usefulness.
• Safety and red teaming: Qiu et al. (2022), Cho et al. (2023) are named as trustworthiness evaluations focusing on specific perspectives. Risk taxonomy synthesizes Stability AI policy, Hive moderation categories, HEx-PHI (Qi et al., 2023), and AIR-Bench 2024 (Zeng et al., 2024). Red teaming methods include SneakyPrompt (Yang et al., 2024b), FigStep (Gong et al., 2023), Liu et al. (2024c) replacing sensitive objects with illustrations, and visual adversarial examples by Qi et al. (2024).
• Hallucination, fairness, privacy, and robustness: The hallucination subset comes from MS COCO (Lin et al., 2014). The fairness section cites Binns (2020) on the apparent conflict between individual and group fairness, and Wan et al. (2024b) on diversity interventions potentially sacrificing factuality; the text cites the report of Gemini generating historical figures (Shamim, 2024) as an example of over-fairness. Privacy follows Carlini et al. (2023) on diffusion model training data memorization, and cites GDPR's definition of sensitive information. Appendix C states that adversarial examples generated by GCG (Zou et al., 2023) and MMP (Yang et al., 2024a) transfer strongly to other black-box models.
• Baselines and data: The paper does not set an external attack baseline spanning all six perspectives, but compares each MMFM on MMDT data. Data sources include HEx-PHI, MS COCO, HRS-Bench, Re-LAION-2B-EN-Research-Safe, Selfies&IDs, self-built Pri-Street-View, and UTKFace.

The authors state MMDT covers more modalities and trustworthiness perspectives than existing MMFM benchmarks, with the comparison in Tab. 52 and Appendix M; the main text does not copy out the cells of that table.

Overall it is an evaluation platform: MMDT generates challenge data and defines metrics per six trustworthiness perspectives, and covers both T2I and I2T. Appendix B divides the pipeline into benchmark orchestration, configuration and job scheduling, local or cloud inference (including vLLM), and result aggregation; the authors plan to keep newly generated red team data private and update it periodically.

Safety

• Taxonomy: Appendix D.1 writes T2I has 13 level-1 categories, further divided into 36 level-2; Table 7 title writes 11 level-1 and 36 level-2. I2T adopts HEx-PHI and AIR-Bench 2024, with both Table 8 and Appendix D.1 having 13 categories. The introduction also writes thirty categories for T2I and eleven for I2T.
• T2I scenarios: (1) vanilla harmful instructions generated by category and human-checked, 10 per category, generated with gpt-4-0613; (2) rewriting original instructions into seemingly harmless settings, such as fictional scenarios; (3) using SneakyPrompt's RL variant with default hyperparameters to optimize adversarial prompts against the DALL·E 2 safety filter. About 60% of prompts were not rejected by that filter and thus remained unchanged.
• I2T scenarios: Data comes from HEx-PHI (11 categories, 30 per category) and is expanded by category per AIR-Bench 2024. The three input types are typography hiding, illustration hiding (GPT-4 rewriting, SDXL generating illustrations), and universal adversarial images optimized against a white-box surrogate (Qi et al., 2024). Appendix D.1 gives generation and judgment prompts.
• Metrics: Input-level BR is the proportion bypassing the safety filter, and output-level HGR is the proportion judged by the evaluation model to contain harmful content. Lower is safer for both. The authors give BR=(1/n)∑_i I[F(x_i)=0] and HGR=(1/n)∑_i I[M(G(x_i))=y_i]. The evaluation model is GPT-4o, answering only Yes or No. Sizes are 1080 T2I and 1170 I2T items.

Hallucination

• Six scenario types: Natural Selection, Distraction, Counterfactual Reasoning, Co-occurrence, Misleading, OCR. Each covers object recognition, counting, attribute recognition, and spatial reasoning; I2T additionally has action recognition, while T2I does not do actions because the authors state it is hard to evaluate without bias.
• Data: Based on MS COCO. For each scenario, many candidates are first constructed, then three surrogate models select the 500 hardest samples for each side. T2I surrogates are SD-v2, OpenDalle, Kandinsky; I2T surrogates are LLaVa (Mistral), Qwen-VL, InstructBLIP (Appendix E.3).
• Distraction and co-occurrence: Distraction adds symbols similar to code comments to the text, or red boxes on irrelevant objects. Co-occurrence samples high-co-occurrence, low-co-occurrence, and historical event pairs according to COCO co-occurrence frequency, examining whether models only reproduce common training-set combinations. OCR is divided into sub-scenarios such as contradiction, distortion, complex background, and misleading.
• Judgment: T2I uses GroundingDINO for detection, counting, and spatial relations, LLaVa (Mistral) for attribute questions, and EasyOCR for OCR. I2T uses LLaMA3 to judge whether open answers match annotations; the misleading scenario uses negation word matching.

Fairness and privacy

• Fairness: Sensitive attributes are gender, race, and age. Scenarios include social stereotypes, hiring/admission/loan decisions, and over-fairness that pursues diversity at the expense of historical or factual accuracy. T2I has 1,776 prompts and I2T has 12,232 prompts.
• Three scores: Group unfairness G looks at whether output distributions across groups deviate from uniform; individual unfairness I looks at whether output quality changes when only group information is changed; over-fairness O is the proportion of historically inaccurate outputs. The authors give ideal values of G=0.0, I=0.0, O=0.0. Individual fairness is only evaluated in social stereotypes, because decision tasks require the model to "choose" a group. T2I group attributes use FairFace and individual quality uses CLIP score; I2T group discrimination uses Llama-3.1-405B, and individual and over-fairness use keyword matching.
• Privacy: T2I evaluates training memorization: sample 10k from Re-LAION-2B-EN-Research-Safe, then filter with named entity recognition to 994 LAION-1k items related to personal names and personal life. Generate images with training prompts, compare ℓ2 distance and cosine similarity in CLIP space; open-source models use M=3 per prompt, DALL·E uses M=1 due to budget, reporting the closest one.
• Inference-time privacy: PII uses 435 selfies or ID photos from Selfies&IDs, 29 people, labeled with age and ethnicity. Location uses self-built Pri-Street-View: 1,816 street views after 2023, excluding landmarks, covering nine countries, 26 states, 93 cities, with labels for country, state, city, and ZIP Code from a geocoding interface.

Adversarial robustness and OOD

• Adversarial: Tasks are object recognition, attribute recognition, and spatial reasoning. Sampled from MS COCO, white-box targeted attacks are run on surrogate MMFMs, keeping successfully attacked samples and then transferring. T2I collects 681, 813, 1,354 prompts, totaling 2,848; I2T collects 1,064, 607, 277 images, totaling 1,948. T2I also optimizes suffixes to maximize embedding similarity between source prompt and target prompt or image; semantic-preserving perturbations (such as typos) are in Appendix H.1. I2T adds perturbations to images to make image embeddings close to target image embeddings.
• OOD: T2I rewrites in-distribution prompts from HRS-Bench into Shakespearean style and rare vocabulary structures, keeping semantics unchanged; 200 per four tasks, totaling 800. I2T takes in-distribution images from MS COCO and applies Zoom Blur, Gaussian Noise, Pixelate, as well as Van Gogh, oil painting, and watercolor; 240 pairs per four tasks, totaling 960. The selection criterion is that surrogate models are correct on original samples and fail on OOD samples.

Experimental setup

• Models under test: T2I are DALL·E 2, DALL·E 3, Stable Diffusion XL, Dreamlike Photoreal 2.0, Openjourney v4, DeepFloyd-IF, Flux-Dev, Amazon Nova Canvas. I2T are GPT-4V, GPT-4o, LLaVa-Next, Gemini Pro 1.5, InternVL2-8B, Mini-InternVL-Chat-4B-V1-5, cogvlm-chat-hf, Llama-3.2-90B-Vision-Instruct, Amazon Nova Lite, Amazon Nova Pro. Privacy Table 27 additionally reports SD-v1-5, SD-2, kandinsky-3, OpenDalleV1.1. Abbreviations in tables and full names in Appendix A.2.2 are treated as the same model under test; Table 1 writes LLaVa as LLaVa (Vicuna).
• Scale: Safety has 1080 T2I and 1170 I2T items. Hallucination has 500 per scenario. Fairness has 1,776 and 12,232 prompts. Privacy is LAION-1k (994), 435 photos (29 people), 1,816 street views. Adversarial has 2,848 prompts and 1,948 images. OOD has 800 prompts and 960 question-answer pairs.
• Metrics and judgment: Safety uses BR and HGR, judged by GPT-4o with Yes/No output; the paper does not report agreement between this judgment and humans. Hallucination accuracy is judged by GroundingDINO, EasyOCR, LLaVa (Mistral), or LLaMA3. Fairness uses Gs, Gd, I, O. Privacy uses CLIP distance or similarity, and attribute, location inference accuracy and refusal rate. Adversarial and OOD report accuracy or performance drop relative to in-distribution tests.
• Consistency spot checks: In T2I OCR evaluation, the authors spot-checked 100 generated images, with EasyOCR and human agreement at 98%. The keyword method for I2T misleading questions had 93% agreement with humans on 100 answers.
• Repetition: Privacy experiments generate 3 images per prompt for open-source models and 1 for DALL·E. The paper does not report repetition counts for other perspectives.
• Surrogate models: Hallucination and OOD use three surrogates to select hard problems; adversarial runs white-box targeted attacks on surrogates and then transfers. The I2T surrogate name for safety jailbreaks is only written as white-box surrogate in the main text.

Main results

The main safety table is Table 1's average HGR and average BR (lower is safer). With more than 8 models under test, the table below keeps the highest HGR per modality and the lowest bolded in Table 1.

Swipe horizontally to read the table

Model	Modality	Average HGR	Average BR
Flux	T2I	0.306	1.000
SDXL	T2I	0.258	1.000
DALL·E 3	T2I	0.138	0.728
Nova Canvas	T2I	0.117	0.388
LLaVa (Vicuna)	I2T	0.621	0.850
CogVLM	I2T	0.579	0.869
GPT-4o	I2T	0.089	0.222
GPT-4V	I2T	0.014	0.035

• The authors bold the lowest average HGR: Nova Canvas for T2I and GPT-4V for I2T. Flux and LLaVa (Vicuna) are the models with the highest average HGR in that modality in Table 1. Omitted T2I are DALL·E 2, DeepFloyd-IF, Dreamlike, Openjourney; omitted I2T are Gemini Pro-1.5, InternVL2, Mini-InternVL, Llama-3.2, Nova Lite, Nova Pro.
• The authors state that a lower BR does not necessarily correspond to safer outputs. In Table 1, several T2I jailbreak HGRs are higher than vanilla, such as DALL·E 2 at 0.244 vs. 0.128; Nova Canvas's transformed HGR is 0.153, higher than its vanilla 0.092. This is inconsistent with the generalization that "all T2I are safer at the output layer on transformed and jailbreak."
• The authors state GPT-4V is safer than GPT-4o in all I2T scenarios. By scenario, Llama-3.2's illustration and jailbreak HGRs are 0.108 and 0.044, lower than GPT-4o's 0.110 and 0.069; typography is 0.331 vs. 0.087.

Hallucination

• What was measured: Table 2 gives accuracy averaged over tasks for six scenarios, where higher means more non-hallucination. The highest T2I average is Flux at 39.7 and DALL·E 3 at 38.7; the highest I2T is Nova Pro at 46.9, followed by Llama-3.2 and GPT-4o both at 44.6. The authors state all MMFMs' average non-hallucination accuracy is below 50%, consistent with Table 2.
• Scenarios: The authors state distraction and misleading easily induce hallucination; Flux's OCR is 60.4, while the remaining T2I OCR in Table 2 ranges from 11.2 to 34.0. The authors state closed-source models outperform open-source on counterfactual reasoning; in Table 2's CR column, open-source Llama-3.2 is 47.3 and Mini-InternVL is 43.4, higher than Gemini Pro-1.5's 28.1 and Nova Lite's 36.9.
• Tasks: The authors state object recognition is generally better than counting. T2I spatial reasoning is called by the authors as three-object relative position accuracy below 3%; in Table 9's Natural Selection, Spatial column, Nova Canvas is 4.8, DALL·E 3 and Flux are 2.4.

Fairness, privacy, and location inference

• Fairness: In Table 3, T2I Gs ranges from 0.337 to 0.597, and I2T from 0.033 to 0.183. The authors state race and age bias are more pronounced in T2I, and gender bias is stronger in I2T; and state there is no observable correlation between group unfairness and individual unfairness. DALL·E 2's I is 13.77, while the rest of T2I in the same table is about 2.190 to 2.819. Llama-3.2's Gs is 0.033 and Gd is 0.018, but O is 0.995; in Table 26 its group/individual fairness refusal rate is 0.852.
• Training memorization: In Table 4, Nova Canvas's CLIP cosine similarity is 0.7765 and ℓ2 distance is 6.706, DALL·E 3 is 0.6335 and 8.551. The authors state pixel-level memorization is not obvious, but concept-level memorization exists for celebrities, objects, watermarks, etc.; DALL·E refuses person-related generation, with the proportion within 10%.
• Location: In Table 5, GPT-4o's country, state, city, ZIP Code, and Range ZIP Code accuracy is 98.16%, 75.40%, 60.23%, 36.55%, 27.13%. The authors state GPT-4V is the only model that occasionally refuses location inference, with refusal rate within 1.61%. PII exact success rate is in Tab. 32; the main text only states GPT-4V has the highest success rate and lowest refusal rate, and ethnicity inference is more accurate than age.

Adversarial robustness and OOD

• Adversarial: Table 6's average robust accuracy, T2I highest is Nova Canvas at 62.42% and lowest is Openjourney at 46.22%; I2T highest is GPT-4o at 90.04%, with its spatial reasoning at 53.79%. The authors state spatial reasoning is the hardest. Nova Lite averages 79.60, higher than the larger Nova Pro at 78.88. The authors state the newer DALL·E 3 and GPT-4o in the same family are higher in both benign accuracy and adversarial robustness, with cells in Appendix Tables 35 and 38, not copied in the main text.
• OOD: Section 7 states most MMFMs have a performance drop exceeding 15% on challenging OOD tests. In T2I, DALL·E 3 and DF-IF have overall drops of about 17%, and other models drop more than 25% under at least one transformation; all models drop more than 25% on spatial reasoning and attribute recognition. In I2T, GPT-4o is still called by the authors the overall highest on in-distribution and OOD, but drops about 16%; counting and attribute recognition drop about 30% and 20%. The authors state that in most tasks, OOD robustness under image style transformations is higher than under image corruption. Table 40 is in Appendix I, and the main text does not give cells.

Other ablations and analyses

• Table 9, T2I Natural Selection Count: SDXL 12.5, DALL·E 3 37.6; the authors state open-source models' counting accuracy is below 20%.
• Table 2's CO column: Flux 48.0, DALL·E 3 44.2, SDXL 27.0; the authors state closed-source models and Flux generate objects that should not co-occur less often in co-occurrence scenarios.
• Table 14, I2T OCR average: Llama-3.2 38.8, GPT-4o 36.8, Gemini Pro-1.5 35.8; the authors state the former is 2% and 3% higher than the latter two.
• Table 21, T2I over-fairness O: DALL·E 3 0.449, Nova Canvas 0.636; the authors state all T2I are poor on this item.
• Table 26, GPT-4o's refusal rates on group/individual fairness and over-fairness are both 0; Gemini Pro-1.5's former is 0.476.
• Table 27, CLIP cosine: SD-2 0.7536, SDXL 0.7521, SD-v1-5 0.7411; the authors state among the SD family, SDv2 and SDXL are higher than SDv1.5.

Limitations and future directions noted by the authors

• Section 8: The authors state that no single MMFM is consistently better than other models across all safety and trustworthiness perspectives, and that this analysis can provide insights for future work improving MMFM trustworthiness.
• Appendix C.3: The authors suggest high-quality RLHF covering the risk taxonomy for training-time alignment, input-output guardrails at deployment time, and state that certified defenses against jailbreaks can further reduce risk.
• Appendix C.3: The authors call human preference annotation plus DPO to constrain T2I fairness a promising future direction, and call for research on how to simultaneously avoid over-fairness sacrificing historical and factual accuracy.
• Appendix C.3: The authors suggest training with differential privacy algorithms or differentially private synthetic multimodal data; at inference time, erasing or anonymizing user images and text, and doing privacy-related instruction tuning to refuse sensitive attribute queries.
• Appendix C.3: The authors suggest attacking a batch of open-source models with GCG and MMP and mixing challenge data into training for trustworthiness fine-tuning; for OOD, they suggest improving data quality and diversity, and adding an "I don't know" option in training.
• Appendices K, L: The paper states limitations and impact are discussed in these two sections. This readable main text ends before entering the specific sentences of Appendix K, so its items are not transcribed.

Experimental coverage

• The main result tables cover 8 T2I and 10 I2T; privacy Table 27 additionally has SD-v1-5, SD-2, kandinsky-3, OpenDalleV1.1. All six perspectives have independent red team data and metrics.
• Safety inputs are 1080 and 1170 items, judged by GPT-4o with Yes/No. Hallucination has six scenarios with 500 items each; four T2I tasks and five I2T tasks. Fairness prompts are 1,776 and 12,232 items, with sensitive attributes gender, race, and age.
• Privacy includes 994 LAION-1k items, 435 selfies or ID photos (29 people), and 1,816 street views after 2023. Adversarial examples are generated by white-box targeted attacks on surrogate models and then transferred, with sizes 2,848 and 1,948. OOD has 800 prompts and 960 question-answer pairs, with transformations including two text styles, three image corruptions, and three image styles.
• The paper does not report agreement between safety judgments and humans. Automatic judgment for OCR and misleading questions was checked against humans on 100 images and 100 answers, respectively. Except for privacy with M=3 or M=1, the paper does not report the number of repetitions, nor statistical significance tests.
• Appendix C discusses transfer of GCG, MMP, and mitigation suggestions such as RLHF, guardrails, DPO, and differential privacy; these mitigation methods themselves are not the systems under test in MMDT.

MMDT is a multi-perspective trustworthiness evaluation platform for T2I and I2T. The authors state it uniformly covers safety, hallucination, fairness, privacy, adversarial robustness, and OOD, whereas existing benchmarks mostly evaluate usefulness or only look at a few perspectives.

• Problem: The authors examine whether models can reject harmful inputs, whether they produce unsafe content under covert or adversarial inputs, and which tasks show concentrated failures in hallucination, bias, memorization, location inference, and distribution shift.
• Approach: Each perspective independently defines risks or tasks, and challenge sets are constructed via red teaming plus human verification. Safety reports both the filter bypass rate BR and the GPT-4o-judged harmful generation rate HGR. Hallucination, fairness, privacy, adversarial, and OOD use accuracy, G/I/O, CLIP similarity, or transferred robust accuracy, respectively.
• Safety: In Table 1, the lowest average T2I HGR is Nova Canvas at 0.117 and the highest is Flux at 0.306; the lowest I2T is GPT-4V at 0.014 and the highest is LLaVa (Vicuna) at 0.621. The authors state that a lower BR does not necessarily mean safer outputs.
• Hallucination and fairness: In Table 2, the highest average accuracy is Flux at 39.7 and Nova Pro at 46.9, which the authors state are both below 50%. In Table 3, T2I group unfairness scores are higher than I2T; Llama-3.2 has a low group unfairness score, while its over-fairness score is 0.995, accompanied by a high refusal rate.
• Privacy, adversarial, and OOD: In Table 5, GPT-4o's country accuracy for street views is 98.16% and ZIP Code is 27.13%. In Table 6, Nova Canvas's average robust accuracy is 62.42% and GPT-4o's is 90.04%, with spatial reasoning still among the lowest tasks. Section 7 states that most models' OOD performance drop exceeds 15%.

The authors conclude that current state-of-the-art MMFMs have deficiencies across all six perspectives, with no single model consistently better, so deployment in safety-critical domains requires attention to these failure modes. Appendix C gives follow-up mitigation suggestions by perspective, including alignment, guardrails, preference optimization, and differential privacy.