Robbing the Fed: Directly Obtaining Private Data in Federated Learning by Modifying the Model
Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Models
Translation in progress — untranslated text is omitted.
Researchers at the University of Maryland and New York University propose a privacy attack against federated learning in which a malicious server makes small modifications to the model it distributes and can then directly recover data from the gradient updates returned by users. Its imprint module consists of a linear layer plus ReLU and uses a data distribution the server can estimate to construct bins, so that structured entries that can be analytically inverted appear in the gradients, without optimization-based inversion. Adding 128 bins before ResNet-18 allows near-perfect recovery of most images from the aggregated update of 64 ImageNet images, with an average PSNR of 75.75; the one-shot variant requires only about 1% additional parameters on ResNet-18 to recover a single image verbatim from the aggregate average of 16384 images. The method does not rely on image priors, and on a transformer language model with 512 bins it fully recovers 110 of 128 sentences. The paper notes that the protection of secure aggregation thus degrades to secure shuffling, that local differential privacy requires relatively large noise to mitigate it, and that the attack code has been open-sourced.
- In federated learning, aggregated gradients are believed to protect user data, and existing honest-but-curious inversion attacks fail at larger batch sizes. The authors ask: if the server maliciously modifies the model architecture and parameters, is aggregation still safe?
- Proposes the imprint module: a linear layer plus ReLU, using a linear function h to measure the data and bin it according to its CDF, so that the gradient of each bin contains only the data falling into that bin, and subtracting the gradients of adjacent bins analytically recovers a single data point without solving an inverse problem.
- With ImageNet batch 64 and 128 bins, average PSNR 75.75; the one-shot variant recovers a single image from an aggregation of 16,384 images (PSNR 161.36); on wikitext, 512 bins recovers 110/128 sentences; under fedAVG multi-step updates, IIP 70.31%.
- The authors point out that local differential privacy noise addition can defend but requires large noise; at σ=0.01, data still visibly leaks; experiments are only conducted on ImageNet, CIFAR-10, and wikitext, and the number of repetitions and cross-model systematic evaluation are not reported.
The paper shows that a malicious server can directly recover user data by slightly modifying the model, bypassing aggregation defenses, and that federated learning deployments relying on large-batch aggregation to protect privacy need to reassess their security boundaries.