ShieldCLIP: Selective Safety Alignment for Harmful Content in Multimodal Foundation Models

The research proposes ShieldCLIP, the first framework to condition safety alignment on each modality's own safety state rather than the sample's source, redirecting only unsafe parts while preserving safe content. The authors also release the ViSUv2 dataset, containing 195k quadruples, covering 578 concepts and 28 categories, with independent safety labels for each modality. Based on these labels, ShieldCLIP defines a four-way conditional objective: anchoring safe content, redirecting unsafe modalities to their corresponding safe versions, updating only the unsafe branch for mixed samples, and enforcing consistency when both modalities are unsafe. On cross-modal retrieval, text-to-image generation with Stable Diffusion v1.4 and SDXL, and image-to-text generation with LLaVA, ShieldCLIP consistently reduces harmful outputs compared to prior safety-aligned encoders and strong mitigation baselines, while preserving utility in the original embedding space. Ablation studies show that both modality-specific supervision and the selective alignment objective contribute.