MultiPriv: A Benchmark for Individual-Level Privacy Inference in Vision-Language Models

The researchers propose the MultiPriv benchmark and claim it is the first benchmark to systematically evaluate the individual-level privacy inference capability of vision-language models (VLMs). The benchmark introduces the Privacy Perception and Reasoning (PPR) framework, constructs a bilingual multimodal dataset containing synthetic personal profiles, associates identifiers such as faces and names with sensitive attributes, and sets up nine tasks including attribute detection, cross-image re-identification, and chain reasoning. A large-scale evaluation of more than 50 open-source and commercial VLMs shows that, in its controlled benchmark, 60% of commonly used VLMs can complete individual-level privacy inference with accuracy up to 80%, which the authors claim poses a significant potential threat to personal privacy. The benchmark is released at https://github.com/CyberChangAn/MultiPriv-PII。