Local Model Poisoning Attacks Against Byzantine-Robust Federated Learning

1. 01What problem does this paper attempt to solve?

   The authors study whether local model poisoning attacks can break federated learning aggregation rules claimed to be resistant to Byzantine faults.

   Core question: can an attacker directly manipulating the local model parameters sent by compromised worker devices raise the test error rate of the global model learned by Byzantine-robust federated learning.

   • Setting and importance: in federated learning, multiple worker devices train collaboratively and a master device aggregates the local models; the machine learning community has proposed aggregation rules such as Krum, Bulyan, trimmed mean, and median, claimed to tolerate Byzantine faults in some devices. The authors argue that the asymptotic error guarantees of these rules are expressed with Õ notation and ignore constants, so they cannot characterize the actual error rate.
   • Shortcomings of existing methods: the authors state that existing poisoning attacks are mainly data poisoning, which only corrupts the training data and assumes the learning process is intact, and have limited success against Byzantine-robust federated learning (experiments in Section 4.4); existing work has demonstrated local model poisoning only on the non-robust mean aggregation rule.
   • Core idea: model the forging of local models in each iteration as an optimization problem, making the global model deviate maximally in the direction opposite to its direction of change without attack; accumulation over multiple rounds makes the learned model differ from that without attack.
   • Paper contributions: the authors state this is the first systematic study of attacks on Byzantine-robust federated learning; they propose local model poisoning attacks and generalize the two data poisoning defenses RONI and TRIM into two defenses, ERR and LFR.
   • Threat model:
     • Goal: untargeted poisoning, making the global model produce a high error rate indiscriminately on test samples, ultimately causing denial of service.
     • Capability: control fewer than 50% of the worker devices (can inject fake devices or compromise benign devices), and can arbitrarily manipulate the local model parameters these devices send to the master device.
     • Knowledge: knows the compromised devices' code, training data, and local models; the aggregation rule is known or unknown in two cases; for benign devices' training data and local models, there are two cases, full knowledge and partial knowledge.
     • Victim systems: federated learning systems using the Krum, Bulyan, trimmed mean, and median aggregation rules; the authors assume the service provider knows the number of compromised devices and sets the aggregation rule parameters accordingly.

   Collapse this answer

2. 02What related research exists?

   Related work is divided into four groups: data poisoning attacks, data poisoning defenses, Byzantine-robust aggregation rules, and other ML security threats.

   Data poisoning attacks

   • Biggio et al. (2012), Muñoz-González et al. (2017), etc. — inject malicious data into training data, targeting centralized machine learning; the authors note these attacks assume the learning process is complete, unlike this paper's attack on the learning process.
   • Muñoz-González et al. (2017)'s back-gradient optimization (BGA) — the authors call it the state-of-the-art untargeted data poisoning attack on multiclassifiers, and this paper compares with it on an MNIST subset.
   • Xie et al. (2019) — an untargeted poisoning attack via inner product manipulation, targeting Krum and median; the authors note it is concurrent work with this paper.

   Data poisoning defenses (centralized)

   • Barreno et al. (2010)'s RONI — measures the impact of each training sample on model error rate and removes samples with large negative impact; Suciu et al. (2018)'s tRONI is its variant for targeted attacks.
   • Jagielski et al. (2018)'s TRIM — jointly solves for the training subset and model parameters, treating samples outside the subset as malicious.
   • The authors note these defenses do not directly apply to local model poisoning, because this paper's attack does not inject malicious data; this paper generalizes RONI and TRIM into ERR and LFR.

   Byzantine-robust aggregation rules and federated learning defenses

   • Krum (Blanchard et al., 2017), Bulyan (Mhamdi et al., 2018), trimmed mean and median (Yin et al., 2018) — claimed to tolerate Byzantine faults of some worker devices, with trimmed mean and median having order-optimal error rates on strongly convex objectives; this paper's attack targets exactly these rules.
   • Fung et al. (2018) — weighted averaging of devices by historical local models; the authors note it can only defend against label flipping attacks, and label flipping can already be defended against by existing Byzantine-robust aggregation rules.

   Other security and privacy threats

   • Adversarial examples (Szegedy et al., 2013, etc.) target the test phase; model inversion, membership inference, attribute inference, model stealing, etc. target confidentiality; the authors state these differ from this paper's attack on training-phase integrity, and privacy risk research in federated learning (e.g., Hitaj et al., 2017) is orthogonal to this paper.

   Baselines and benchmarks: the comparison attacks are Gaussian attack, label flipping attack, BGA; the datasets are MNIST, Fashion-MNIST, CH-MNIST, Breast Cancer Wisconsin (Diagnostic). The authors position this paper as: the first systematic attack study targeting Byzantine-robust federated learning by directly manipulating local model parameters (per the authors).

   Collapse this answer

3. 03How does the paper address this problem?

   Model the forged local models as an optimization problem for each round, maximizing the global model's deviation in the opposite direction, and solve it separately for Krum and for trimmed mean/median.

   Overall idea: in each iteration the attacker solves an optimization problem to forge the local models of the compromised devices, so that the aggregated global model deviates maximally in the direction opposite to its direction of change without attack; the paper calls this method local model poisoning attacks.

   Problem formalization

   • Let s be the direction vector of the change in global model parameters without attack (each parameter takes 1 or −1), w the global model without attack (before-attack global model), and w0 the global model after attack.
   • The core optimization problem is max_{w′1,⋯,w′c} s^T (w − w′), with the constraint that w and w0 are obtained by applying the aggregation rule A to the local models before and after the attack, respectively; the meaning is to forge c compromised local models so that the global model's directed deviation in the opposite direction is maximized.
   • The authors state they also considered a deviation goal without direction, and experiments show the directed deviation goal is better on trimmed mean and median (Appendix B).

   Attacking Krum

   • Two approximations: set w′1 = wRe − λs (wRe is the global model received in this round, λ > 0), and make the other c − 1 compromised local models within distance ε of w′1 to "support" w′1 being selected by Krum; the authors state the approximation yields a suboptimal solution.
   • Solving for λ: Theorem 1 gives an upper bound on λ, based on which binary search is used, searching until no solution is found once λ falls below the threshold 1 × 10−5.
   • partial knowledge: estimate the direction of change s̃ using the mean of the compromised devices' before-attack local models, treating the compromised devices' before-attack models as benign models; when no solution is found, iteratively add forged local models w′2, etc.; the authors state this makes the attack more effective against Krum under partial knowledge.

   Attacking trimmed mean and median

   • full knowledge: for each parameter dimension j, if s j = −1 take c values greater than wmax, j, otherwise take c values less than wmin, j; in implementation, randomly sample in intervals such as [wmax, j , b · wmax, j ] or [wmin, j /b, wmin, j ], with b = 2; the authors state that when b > 1 the attack does not depend on the specific value of b; median reuses the same attack.
   • partial knowledge: estimate the direction using the mean µ j and standard deviation σ j of the compromised devices' parameters, and assume benign devices' parameters follow a Gaussian distribution, sampling in [µ j + 3σ j , µ j + 4σ j ] or [µ j − 4σ j , µ j − 3σ j ]; the authors state that under this assumption the forged parameters exceed the benign extremes with high probability (e.g., 0.898–0.998 when m = 100, c = 20), and the attack remains effective when the actual distribution is not Gaussian.

   Defenses: ERR, LFR and Union

   • Setting: the master device holds a small validation dataset and knows the upper bound c on the number of compromised devices; before each round of aggregation it first removes suspected malicious local models.
   • ERR: for each local model, compare the error rate of the global model on the validation set with and without including it, and remove the c with the largest impact on error rate (generalized from RONI).
   • LFR: likewise compare with and without inclusion, but using cross-entropy loss, and remove the c with the largest impact on loss (generalized from TRIM); Union removes models removed by either ERR or LFR.
   • The authors note that these defenses make the global model learn and adapt to new data more slowly.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On 4 datasets with LR and DNN, attacks are run against 4 aggregation rules, mostly raising the error rate, and transferability, comparison with BGA, and the ERR/LFR defenses are also evaluated.

   Experimental Setup

   • Datasets: MNIST and Fashion-MNIST (60,000 training / 10,000 test each, 10 classes), CH-MNIST (5,000 histology images, 8 classes, 4,000 training / 1,000 test), Breast Cancer Wisconsin (Diagnostic) (569 cases, 30 features, binary classification, 455 training / 114 test).
   • Classifiers: multiclass logistic regression (LR, MNIST only); DNN (MNIST, Fashion-MNIST, Breast Cancer use the architecture in Table 7a, CH-MNIST uses ResNet20).
   • Default parameters (Table 1): m = 100 worker devices, c = 20 compromised, non-IID degree p = 0.5, Krum attack ε = 0.01, trimmed mean β = c, b = 2; LR trained for 500 rounds, DNN for 2,000 rounds, batch size 32 (64 for Fashion-MNIST).
   • Comparison attacks: Gaussian attack, label flipping attack, BGA (compared separately on a 6,000-sample MNIST subset).
   • Metric and repetitions: test error rate; each experiment repeated 50 times and averaged, the authors state the variance is small and therefore omitted.
   • Defense evaluation: validation set is 100 randomly drawn test samples, partial knowledge attack, MNIST + LR.

   Main Results

   Table 2 gives the test error rates of each attack on the four datasets (default settings, Partial/Full are the two knowledge cases of this paper's attack):

   Swipe horizontally to read the table

   Setting	Aggregation rule	NoAttack	Gaussian	LabelFlip	Partial	Full
   LR, MNIST (2a)	Krum	0.14	0.13	0.13	0.72	0.80
   LR, MNIST (2a)	Trimmed mean	0.12	0.11	0.13	0.23	0.52
   LR, MNIST (2a)	Median	0.13	0.13	0.15	0.19	0.29
   DNN, MNIST (2b)	Krum	0.11	0.10	0.10	0.75	0.77
   DNN, MNIST (2b)	Trimmed mean	0.06	0.07	0.07	0.14	0.23
   DNN, MNIST (2b)	Median	0.06	0.06	0.16	0.28	0.32
   DNN, CH-MNIST (2d)	Krum	0.29	0.30	0.43	0.73	0.81
   DNN, Breast Cancer (2e)	Krum	0.03	0.04	0.14	0.17	0.17

   (Fashion-MNIST rows and the trimmed mean/median rows on CH-MNIST and Breast Cancer are omitted; on Fashion-MNIST Krum is 0.16→0.90/0.91, per Table 2c.)

   • The authors state this paper's attack is effective and outperforms existing attacks; Krum is more vulnerable than trimmed mean and median, but on Breast Cancer Krum is comparable to median.
   • The authors suggest the possible reason trimmed mean and median are more robust is that they aggregate multiple local models, whereas Krum selects only one.
   • The authors observe that error rates may vary with data dimensionality: CH-MNIST (4096 dimensions) has higher error rates in most cases, Breast Cancer (30 dimensions) lower in most cases.

   Transfer and Bulyan

   • Unknown aggregation rule (Table 4, MNIST, LR, partial knowledge): the Krum-based attack raises the error rate from 0.12 to 0.15 for trimmed mean and from 0.13 to 0.18 for median; the trimmed mean-based attack cannot transfer to Krum (0.14), and raises median from 0.13 to 0.20.
   • Bulyan (Table 7b, MNIST, LR, m = 100, c = 20, θ = m − 2c, γ = θ − 2c): the partial knowledge attack raises the error rate from 0.14 to 0.36, and full knowledge to 0.38; the authors state the attack on Krum can transfer to Bulyan.

   Comparison with BGA Data Poisoning

   • Setup (Section 4.4): 6,000 training samples uniformly drawn from MNIST, 10-class LR, 100 devices, 20 compromised; BGA runs for 10 days on a GTX 1080Ti GPU to generate 240 (4%) poisoned samples, allocated in two ways, SingleWorker and Uniform.
   • Results (Table 5): BGA has limited effect on Byzantine-robust rules (e.g., on Krum SingleWorker 0.24, Uniform 0.25, NoAttack 0.23), and on the mean rule with Uniform raises it from 0.10 to 0.15; this paper's partial knowledge attack reaches 0.54 on mean and 0.85 on Krum.
   • The authors state they verified that BGA poisoned data is effective in a centralized setting (LR error rate 0.10→0.16), but in federated learning it can only be injected on compromised devices, so its success against robust rules is limited.

   Defense Evaluation

   • Results (Table 6, MNIST, LR, partial knowledge): LFR is comparable to or better than ERR, e.g., when the attack is based on trimmed mean, Trimmed mean + LFR is 0.12 and Trimmed mean + ERR is 0.21; Union is comparable to LFR in most cases.
   • Effective cases: Trimmed mean + LFR (or + Union) has the same error rate as no attack (0.12) when the attack is based on trimmed mean.
   • Ineffective cases: when the attack is based on Krum, the error rate under Krum + LFR still rises from 0.14 to 0.58; the authors state new defenses are needed.
   • Negative results: in some cases deploying a defense instead raises the error rate (e.g., Median + ERR under a trimmed mean-based attack is 0.25, higher than 0.19 without defense), which the authors explain as the defense possibly mistakenly removing benign local models.

   Other Ablations and Analyses

   • Compromised device proportion (Figure 2, MNIST): attack error rate rises with the proportion; the exception is Krum, where the error rate decreases from 5%→10% (LR) and 10%→15% (DNN), which the authors suspect is related to Krum selecting only one model per round.
   • non-IID degree (Figure 3, MNIST): the error rate of each attack rises with p, but the error rate of the attack on Krum fluctuates.
   • Number of SGD rounds (Figure 4a, MNIST, LR, median, partial knowledge): error rate decreases as rounds increase, and at 10 rounds the attack still raises the error rate by more than 30% (per the authors).
   • Total number of devices (Figure 4b): more devices make the attack more effective; the authors state the direction estimation accuracy of partial knowledge against trimmed mean (DNN, MNIST) rises from 72% with 50 devices to 76% with 100 devices.
   • Number of devices sampled per round (Figure 4c): the attack is effective when the compromised proportion mc exceeds 10%–15%.
   • β of trimmed mean (Figure 5a): the larger β is, the weaker the attack, and the no-attack error rate also rises slightly.
   • ε of the Krum attack (Figure 5b): the smaller ε is, the larger the error rate.
   • Proportion of poisoned iterations (Figure 5c): more poisoned iterations give a higher error rate.
   • Alternative training strategy (Table 3, DNN, MNIST): when selecting the global model with the lowest test error rate, the partial knowledge attack still raises Krum from 0.10 to 0.69 and median from 0.06 to 0.32.
   • deviation goal comparison (Table 8, MNIST, LR): the directed deviation goal is higher on trimmed mean (0.52 vs 0.10) and median (0.29 vs 0.12), and both are high on Krum (0.80 vs 0.87).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors state the work is limited to untargeted attacks and that new defenses are needed; the experiments cover 4 datasets, LR and DNN, and 4 aggregation rules.

   Limitations and future directions noted by the authors

   • The work is limited to untargeted poisoning attacks; studying targeted poisoning attacks against federated learning is a future direction (Section 7).
   • Designing new defenses is valuable future work, for example new methods to detect compromised local models and new adversarially robust aggregation rules (Section 7).
   • The authors state that the generalized defenses (LFR, Union) are effective in some cases and not sufficiently effective in others, for example ineffective against Krum-based attacks, and new defense mechanisms are needed (Section 5, Section 7).
   • The authors note that the full knowledge setting has limited applicability in practice, because federated learning's training data is distributed across many devices; this setting is used to estimate an upper bound on the attack threat (Section 2.3).
   • The authors note that the ERR/LFR defenses make the global model learn and adapt to new data more slowly, because new data may be identified as coming from potentially malicious local models (Section 5).

   Scope of experiments

   • The aggregation rules tested are Krum, trimmed mean, and median; Bulyan is evaluated only in Appendix A with a single setting, MNIST + LR; the classifiers are LR (MNIST only) and DNN (four datasets, with ResNet20 for CH-MNIST).
   • The datasets are MNIST, Fashion-MNIST, CH-MNIST, and Breast Cancer Wisconsin (Diagnostic), 4 in total; the defaults are m = 100, c = 20, p = 0.5, and each experiment is repeated 50 times and averaged.
   • The comparison with BGA is conducted on a 6,000-sample subset of MNIST with a 10-class LR; the authors state that BGA cannot scale to the full MNIST.
   • Defense evaluation covers only the three defenses ERR, LFR, and Union, and is conducted only under the setting of MNIST + LR, partial knowledge, and 100 validation samples (Table 6).
   • The paper does not report the computational cost or number of queries of the attack; the parameter-influence experiments in Figures 2–5 are conducted only on MNIST (Figures 4 and 5 use LR + median or the corresponding rule).

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors propose local model poisoning attacks against Byzantine-robust federated learning; error rates rise on four datasets, and the generalized defenses are only partially effective.

   An attack paper targeting federated learning (USENIX Security 2020): the authors study whether an attacker manipulating the local model parameters sent by compromised worker devices can break aggregation rules claimed to be resistant to Byzantine faults.

   • Problem: existing data poisoning attacks only corrupt the training data and have limited effect on Byzantine-robust aggregation rules such as Krum, trimmed mean, and median; the authors state that the asymptotic error guarantees of these rules cannot characterize the actual error rate.
   • Method: model the forging of local models in each round as an optimization problem, making the global model deviate maximally in the direction opposite to its direction of change without attack; solve it for Krum using two approximations plus binary search, and for trimmed mean/median sample near the benign parameter extremes along each dimension; distinguish two knowledge settings, full and partial knowledge. On the defense side, generalize RONI and TRIM into ERR, LFR, and their combination Union.
   • Main results: under the default setting (m = 100, c = 20, p = 0.5), on MNIST with DNN + Krum the error rate rises from 0.11 to 0.75 (partial knowledge, Table 2b); on Fashion-MNIST, Krum rises from 0.16 to 0.90 (Table 2c); on CH-MNIST, Krum rises from 0.29 to 0.73 (Table 2d).
   • Comparison and transferability: label flipping and Gaussian attacks are of limited effectiveness; BGA has limited effect on robust rules on the MNIST subset, whereas the attack in this paper reaches 0.85 on Krum (Table 5); the Krum-based attack transfers to trimmed mean, median, and Bulyan (0.14→0.36, Table 7b).
   • Defenses: LFR outperforms ERR; Trimmed mean + LFR defends against the trimmed mean-based attack (0.12), but the Krum-based attack under Krum + LFR still raises the error rate from 0.14 to 0.58 (Table 6).
   • Authors' conclusion: the authors state that these federated learning methods claimed to be robust are in practice vulnerable to the attack in this paper, and new defense mechanisms are needed; the work is limited to untargeted attacks, with targeted attacks and new defenses left for future work.

   Collapse this answer

Related papers