SafeGen-Bench: A safety benchmark for image-conditioned text-to-video generation

The researchers introduce SafeGen-Bench, a benchmark specifically for evaluating the safety of image-conditioned text-to-video (T2V) models, addressing the gap that existing benchmarks mainly test with malicious text prompts and overlook that combinations of text and images can still generate harmful videos. The benchmark defines 10 malicious categories, covering risks related to temporal sequences and the depicted behaviors, and uses starting frames selected from diverse image and video sources paired with corresponding text prompts to simulate real inputs. Evaluations of various conditional T2V models show that existing models struggle to consistently avoid generating malicious content, with unsafe scores reaching as high as 44.5, especially when high generation quality is required. Tests of text-based and image-based guardrails find that unimodal guardrails are insufficient to provide robust defense, with failure rates of 80% on seven malicious categories. The authors hope the benchmark will promote the development of safer and more controllable conditional T2V models.