When the curious abandon honesty: federated learning is not private
When the Curious Abandon Honesty: Federated Learning Is Not Private
Translation in progress — untranslated text is omitted.
Researchers propose a new data reconstruction attack against federated learning: a malicious central server can make inconspicuous modifications to the weights (trap weights) before distributing the shared model, and thereby extract training data with zero error from the gradients returned by users, with almost no computational cost. The attack scales components of the weight matrix so that typically only one data point in a mini-batch can activate a given neuron, thereby amplifying the data leakage inherent in gradients; with a mini-batch of 100 data points, over 50% of ImageNet training data and over 65% of IMDB sentiment analysis training data are perfectly reconstructed, and it remains effective when all data come from the same class. Even without tampering, gradients of randomly initialized models also passively leak data points: under the same setting, about 26% of CIFAR10 and about 22% of ImageNet can be perfectly extracted; compared with prior attacks that rely on costly optimization or on modifications that are easy to detect, this method applies to fully connected and convolutional networks and also holds under multi-round local training in FedAvg. The paper has been accepted to IEEE Euro S&P.
- Federated learning is often promoted as privacy-preserving because data does not leave the device, but prior work has shown that gradients leak user data. The authors focus on: whether an active and dishonest central party can efficiently and precisely extract individual training data points from large mini-batch gradients by manipulating the shared model weights.
- The authors propose trap weights: the central party sets half of the weights in each row of the first fully connected layer to negative and scales the positive components by a factor s<1, so that each neuron is with high probability activated by only one data point in the mini-batch; the gradient is then projected back to the input domain to read out the data directly, without optimization; extensions to CNNs and embedding models are also given.
- On MNIST, CIFAR10, ImageNet, and IMDB, with B=100 and 1000 neurons, the active attack extracts 54.0%, 54.0%, 45.7%, and 65.4% of data points with zero error, more than about twice that of the passive attack; it remains effective against FedAvg with multiple rounds of local training and non-IID single-class data; locally averaging multiple mini-batches reduces MNIST's R from 54.0% to 2%.
- The authors point out that defenses are either costly or target only this attack, and recommend combining with DP by default; trap weights relies on ReLU fully connected layers, and lossy layers such as pooling and dropout reduce extraction fidelity; experiments are limited to image and text classification and the listed architectures, and the code was planned to be open-sourced after review at the time of writing.
A malicious central server can use trap weights to reconstruct more than half of users' training data with zero error from a large batch of 100 samples at near-zero cost, undermining the common assumption that federated learning protects privacy.
Related papers
- Don't Forget to Sign the Gradients!DefensesarXiv