Privacy-Preserving Model Transcription: Differentially Private Synthetic Distillation

The researchers propose “privacy-preserving model transcription,” a scheme that converts a pretrained teacher model into a privacy-preserving student model without access to private data. Its differentially private synthetic distillation lets the generator, teacher, and student optimize alternately: the teacher and student generate noise-perturbed differentially private labels for synthetic data, the student updates with the noisy labels and acts as a discriminator to drive generator training. The authors prove that the method has differential privacy and convergence guarantees, and experiments show that it outperforms 26 SOTA methods.