Contrastive Privacy: A Semantic Method for Measuring the Privacy of AI Redaction

The paper proposes contrastive privacy, formalizing the privacy evaluation of AI image and text redaction as a quantifiable semantic test: semantic distance models such as CLIP are used to compare the redaction result under audit with versions of reference data before and after redaction; if the result is closer to the unredacted reference, privacy may still be leaked, and there is no need to enumerate specific privacy attributes or provide item-by-item ground-truth labels. The authors evaluated 34 image redaction configurations, 15 social media text models, 1 entity recognizer, and 4 off-the-shelf PII redaction tools on Enron emails, and all settings detected residual semantic associations, including the four PII tools even when all candidates returned by their detectors were removed. Supplementary research showed that fine-tuning on synthetic identities can reveal associations missed by base models, and tests adapted after broader redaction no longer detected them; on 94 matched image sets redacted to hide Leonardo DiCaprio's identity, the evaluations of three models were largely correlated but sometimes disagreed on which redaction was more private, and the authors accordingly recommend using multiple models together.