LIBERO-Safety: a comprehensive benchmark for physical and semantic safety of vision-language-action models

The researchers propose the LIBERO-Safety benchmark for evaluating the physical and semantic safety of vision-language-action (VLA) models. The benchmark procedurally generates randomized safety-critical scenarios in a parameterized manner and develops a key-pose-driven data generation pipeline to overcome the scaling bottleneck of manual teleoperation, thereby constructing a demonstration dataset of 19664 trajectories that are strictly collision-free and contain extensive domain randomization. The team conducted cross-paradigm evaluations of eight VLA models and two embodied foundation models, revealing a tension between generalization and safety: although high-diversity training helps produce safer trajectories, task success rates remain fundamentally constrained by suboptimal trajectory synthesis and semantic misalignment. The authors describe this work as providing a scalable pipeline, dataset, and insights into failure modes for developing safe and reliable VLA models.