Privacy Auditing of Private Prediction
Auditing Private Prediction
Translation in progress — untranslated text is omitted.
The authors propose the first framework for auditing privacy leakage of private prediction, targeting machine learning that enforces privacy protection at the inference stage, and establish empirical leakage lower bounds by constructing adversaries with different poisoning and query capabilities. The framework introduces a new technique for empirically evaluating privacy leakage with Renyi DP, and evaluates four private prediction algorithms: PATE, CaPC, PromptPATE, and Private-kNN. Experiments show that the privacy analysis of private prediction can be improved, that algorithms more susceptible to poisoning leak more privacy, and that leakage when the adversary has no query control is significantly lower than with full query control.
- Differential privacy gives an upper bound on privacy leakage, and auditing gives a lower bound, but existing audits target only DP training; private prediction, which adds noise only at the inference stage, has no auditing technique, and the authors aim to fill this gap.
- The authors propose an auditing framework: a crafter constructs neighboring datasets differing by one data point, the private prediction algorithm produces an interface, and a distinguisher selects queries and compares output distributions; they use the 2-cut of Renyi divergence for per-query RDP auditing with lossless composition, and additionally give the computation of exact Renyi divergence between neighboring histograms for noisy argmax as an upper-bound reference.
- An audit of PATE, CaPC, PromptPATE, and Private-kNN shows: exact computation is below the data-dependent theoretical value, and the authors state that the analysis has slack; algorithms that are easy to poison (PromptPATE, CaPC) leak more; Pois-MINQ, which can only issue natural queries, leaks far less than an adversary that can control queries.
- The authors state that the audit can be improved by better statistical techniques and by poisoning techniques that can influence multiple queries; evaluating the exact divergence over all neighboring histograms is computationally infeasible; the experiments use 200–250 teachers and a parameterized assumption to simulate training randomness, and the paper does not report utility (accuracy) data.