Skip to content

Cronus: Robust Heterogeneous Collaborative Learning Based on Black-Box Knowledge Transfer

Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer

arXiv· arXiv:1912.11279· Hongyan Chang, Virat Shejwalkar, Reza Shokri, Amir HoumansadrNational University of Singapore; University of Massachusetts AmherstIndexed by this site · 23 days ago· Originally publishedDefenses

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

The authors propose Cronus: parties in collaborative learning share only predictions on public data rather than parameters; the authors state that its accuracy drop under poisoning attacks is at most 2.1%, and its membership inference risk is close to random guessing.

Threat modelTwo types of threat: (1) poisoning attack: the attacker controls an ǫ fraction of the n parties (taking the maximum value below the breaking point of each aggregation algorithm), possesses data from the same distribution as benign parties, and can construct malicious updates; (2) membership inference attack: the server or a participating party performs passive/active inference on a single party's update or the aggregated update, and the attacker knows 50% of the target party's members and an equal number of non-members.

Questions
Sharing parameters in federated learning has three problems: susceptibility to poisoning attacks, leakage of training data privacy through parameters, and the requirement that all parties' model architectures be identical. The authors consider sharing parameters the most naive and also most dangerous way of exchanging information in collaborative learning.
Methods
Propose Cronus: Each party first trains independently on its local data, after which it shares only its local model's predictions (soft labels) on unlabeled public data; the server aggregates the predictions with a dimension-independent robust mean estimation algorithm, and each party then fine-tunes its local model using its private data plus the soft-labeled public data, without overwriting parameters.
Experiments and results
Evaluated on SVHN, MNIST, Purchase and CIFAR10: the authors state that all existing federated aggregation algorithms can be broken by some poisoning attack, whereas Cronus's accuracy drop is at most 2.1%; membership inference risk is close to random guessing; and it supports collaboration across heterogeneous architectures.
Limitations and future work
The authors note that under the benign setting Cronus's accuracy is slightly lower than FedAvg (e.g., 80.1% vs. 88.4% on CIFAR10); user-level DP cannot train a good model when there are few participating parties; existing DP algorithms have poor utility. The experiments are limited to 4 image/tabular datasets and at most 32 benign parties.
Experimental setupCNN (SVHN)、FC (MNIST) et al. · SVHN、MNIST et al. · Robustness(攻击下精度/良性精度)、成员推断攻击准确率 et al.
Evaluated models
CNN (SVHN)FC (MNIST)FC (Purchase)DenseNet (CIFAR10)A1A2A3A4A5
Benchmarks
SVHNMNISTPurchaseCIFAR10
Metrics
Robustness(攻击下精度/良性精度)成员推断攻击准确率分类准确率
AI summaryCronus is a robust collaborative (federated) learning framework that replaces parameter sharing with black-box knowledge transfer between local models, controls and substantially reduces the dimensionality of exchanged information, and supports heterogeneous model architectures. The paper evaluates the performance of existing federated learning algorithms under poisoning attacks and states that Cronus is the only secure method. Black-box processing reduces model information leakage, can be combined with existing privacy-preserving algorithms, has significantly lower sample complexity, and its security does not depend on the number of participants.

Cronus is a robust collaborative (federated) learning framework that replaces parameter sharing with black-box knowledge transfer between local models, controls and substantially reduces the dimensionality of exchanged information, and supports heterogeneous model architectures. The paper evaluates the performance of existing federated learning algorithms under poisoning attacks and states that Cronus is the only secure method. Black-box processing reduces model information leakage, can be combined with existing privacy-preserving algorithms, has significantly lower sample complexity, and its security does not depend on the number of participants.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Address the three major problems caused by sharing parameters in federated learning: poisoning vulnerability, privacy leakage, and architectural homogeneity.

    Core problem: Federated learning exchanges knowledge by sharing model parameters, which exposes collaborative learning simultaneously to poisoning attacks, privacy leakage, and architectural homogeneity constraints; the authors consider sharing parameters the "most naive" way of exchanging information.

    • Problem setting: Collaborative (federated) learning lets multiple parties train models without sharing private data, and is applied in sensitive domains such as healthcare, finance, and virtual assistants. But existing algorithms are not resistant to adversarial updates and backdoor attacks, and they leak information about local data.
    • Shortcomings of existing methods: The authors state that all existing defenses merely replace the aggregation function with a robust mean while leaving the rest of the framework unchanged; the theoretical error bounds and sample complexity of these aggregation algorithms all depend on the parameter dimension, and under high-dimensional models the error bounds are too high (the authors call this the "curse of dimensionality"), and parameter-overwriting updates make the model easy to break with small, hard-to-detect changes.
    • Core observation: Sharing parameters is not the only way to exchange knowledge; knowledge transfer can convey knowledge through the model's predictions on public data, reducing the dimension of the exchanged vector from the parameter scale to the output scale.
    • Paper proposes: Cronus—a robust collaborative learning framework that extracts, aggregates, and transfers the knowledge of local models in a black-box manner; it also systematically evaluates the robustness of all existing federated aggregation algorithms under poisoning attacks, and proposes a new attack, OFOM, targeting MWU-type aggregation.
    • Threat model (Section II):
      • Poisoning attack: The goal is to indiscriminately damage the final model accuracy; the attacker controls an ǫ fraction of the n parties and possesses data with the same distribution as the benign parties, allowing it to estimate the benign update distribution; in experiments, ǫ is set to the maximum value below the breaking point of each aggregation algorithm.
      • Membership inference attack: The server or a participating party is the attacker, performing passive observation or active (gradient ascent) attacks on a single party's updates or the aggregated updates; the attacker knows 50% of the target party's member samples plus an equal number of non-members, and trains a binary classifier to distinguish members from non-members.
  2. What related research exists?

    Related work is divided into four groups: federated learning and poisoning attack/defense, robust mean estimation, differentially private training, and knowledge transfer.

    Federated learning and poisoning attacks

    • McMahan et al. [35], Shokri and Shmatikov [47] — proposed federated learning/privacy-preserving deep learning, in which training is collaborative through shared gradients or parameters.
    • Bagdasaryan et al. [10] — a single malicious party launches a targeted poisoning (backdoor) attack on FedAvg; Bhagoji et al. [12] — a single malicious party launches targeted attacks on two robust aggregators, Krum and coordinate-wise median.
    • Baruch et al. [11] — the LIE attack, which bypasses robust aggregators such as Bulyan and Krum; the paper reproduces and uses this attack in its experiments.

    Robust aggregation and robust mean estimation

    • Yin et al. [57] — analyzes median and trimmed-mean aggregation and gives robustness guarantees; Xie et al. [55], Su [51], Alistarh et al. [5] — median/geometric-median-type robust aggregation.
    • Blanchard et al. [13] — Krum, a computable approximation of the geometric median; El Mhamdi et al. [37] — Bulyan, a two-step meta-aggregation of Krum plus trimmed median. The paper notes that these algorithms have been shown to be sensitive to targeted or untargeted poisoning attacks on deep learning models [11], [12].
    • Diakonikolas et al. [16], [17], [15], Li [33] — robust mean estimation, tightening error bounds and reducing dimension dependence; Cronus directly adopts the algorithm of [17] (sample complexity Θ(d log d)).

    Differential privacy and privacy protection

    • Abadi et al. [4] — DP-SGD (record-level DP); McMahan et al. [14] — user-level DP federated learning, which the paper notes requires a large number of participants (5,000 parties per round) to reach usable accuracy, and does not protect against the aggregation server.
    • Treux et al. [52] — a hybrid scheme combining DP with secure multi-party computation; the paper notes that such work is not resistant to poisoning and has a poor privacy-utility tradeoff [27].
    • Nasr et al. [39], [40] — adversarial regularization training for membership privacy and active/passive white-box inference attacks, whose attack models and defense mechanisms the paper uses for evaluation; Melis et al. [36] — feature leakage attacks in collaborative learning.

    Knowledge transfer

    • Hinton et al. [21], Ba and Caruana [9] — knowledge distillation; Anil et al. [7], Jeong et al. [28] — co-distillation, which the paper notes does not defend against poisoning and inference attacks.
    • Papernot et al. [43], [44] — PATE, which labels public data with an ensemble of teachers and trains a student with DP; the paper notes that the PATE setting is fundamentally different from Cronus: in PATE, both the teachers and the noise addition are performed by a trusted entity that holds all the data.

    Baselines and benchmarks: the comparison baselines are federated learning with 6 aggregation algorithms (Mean/FedAvg, Median, MwuAvg, MwuOpt, Bulyan, Krum), and the benchmarks are SVHN, MNIST, Purchase, CIFAR10. The authors position this work as: the only collaborative learning framework that replaces parameter sharing with black-box prediction sharing, thereby simultaneously obtaining tight robustness guarantees, low inference risk and heterogeneous support.

  3. How does the paper address this problem?

    Cronus replaces parameters with predictions on public data (soft labels) as the object of exchange, together with a dimension-independent robust mean aggregation.

    Overall idea: Replace the information exchanged in collaboration from model parameters to the model's prediction vectors on unlabeled public data, reducing the aggregation dimension from the parameter scale to the number of classes, so that a robust mean estimation algorithm with tight error bounds can be used, while eliminating the white-box inference attack surface and supporting heterogeneous architectures. The framework is named Cronus.

    Problem setting

    • Each of the n parties holds a private training set Di, and there is additionally an unlabeled public dataset Xp (a set of feature vectors) shared by all parties.
    • Unlike FedAvg, Cronus does not overwrite the global model onto the local models: each party updates its local model independently, and the aggregated knowledge serves only as an additional training signal.

    Two-stage training procedure (Algorithm 5)

    1. Initialization stage: Each party independently trains on its local data Di for T1 rounds with no collaboration; then it computes the initial predictions Yi0 of its local model on Xp and sends them to the server.
    2. Collaboration stage (T2 rounds, three steps per round): Each party computes soft labels Yi on Xp with its local model and uploads them; the server aggregates the predictions separately for each public sample, Ȳ = fCronus(Y1, ..., Yn), and broadcasts the result; each party updates its local model using Di ∪ (Xp, Ȳ).
    • Key parameters: SVHN/MNIST/Purchase have 50 rounds in each of the two stages; CIFAR10 has 200 rounds in total (Appendix A).

    Robust aggregation fCronus (Algorithm 6)

    • It adopts the robust mean estimation algorithm of Diakonikolas et al. [17], aggregating the n prediction vectors of each public sample separately.
    • Intuition: If the empirical mean is contaminated, the empirical variance in the contamination direction is much larger than the population variance; the algorithm iteratively finds the maximum-variance direction v*, projects and filters out the portion of samples farthest from the mean, until the variance in every direction is not large (λ* ≤ 9).
    • Theoretical guarantee (Theorem 1): With sample size Θ((d/ǫ) log d), it outputs, with probability 9/10, an estimate satisfying ||μ̂ − μP||2 = O(σ√ϵ), i.e., the error is independent of the dimension d—this is Cronus's core advantage over parameter aggregation (whose error contains a d factor).
    • Practical simplification: Deterministically filter an ǫ/2 fraction of the inputs each round, for a total of 2 filters.

    Privacy and heterogeneity design

    • Sharing only predictions makes white-box inference attacks entirely impossible; distillation on data that does not overlap with the training set itself reduces leakage [48].
    • The black-box setting allows each party to locally use membership privacy mechanisms (adversarial regularization [39]) or prediction privacy mechanisms [18].
    • Parties only need to have consistent output dimensions (the same task); architectures can be heterogeneous, and may even belong to different model families.

    Attack design (for evaluation)

    • Reproduce three existing poisoning attacks: Label flip (malicious parties consistently flip local labels), PAF (malicious updates take the benign mean plus an arbitrarily large vector), and LIE [11] (constructs small perturbations using the mean and variance of benign updates).
    • Propose the OFOM attack: Targeting distance-weighted MWU-type aggregation (MwuAvg, MwuOpt), two malicious parties respectively submit an arbitrarily far update and an update located between the benign mean and that far point, so that after the first round the malicious parties' weights approach 1 and the benign parties' weights become negligible; only two malicious parties are needed.
    • Membership inference attack: Passive attacks directly observe updates; active attacks (gradient ascent [40]) have the server perform gradient ascent on target samples to widen the loss gap between members and non-members.
  4. What experiments did the paper conduct?

    Evaluate robustness, membership inference risk, DP cost, heterogeneous collaboration, and communication overhead on 4 datasets.

    Experimental setup

    • Datasets and participants (Table III): SVHN (32 benign parties, 5,000 samples each, CNN), MNIST (28 parties, 2,000, FC), Purchase (16 parties, 10,000, FC), CIFAR10 (16 parties, 2,500, DenseNet-100); public data is 10,000 in all cases.
    • Comparison methods: stand-alone, centralized, FedAvg, and federated learning with six aggregators Mean/Median/MwuAvg/MwuOpt/Bulyan/Krum, compared against Cronus.
    • Number of malicious parties (Table V): set to the maximum value below each aggregator's breaking point, e.g., on SVHN, Mean 1, Median/MWU/Cronus 31, Krum 29, Bulyan 9.
    • Metrics: robustness = accuracy under the strongest attack / benign accuracy; membership inference risk = inference attack accuracy (half members, half non-members; 50% is random guessing).
    • Membership inference evaluation: uses the white-box inference model of [40] (taking features such as last-layer gradients and outputs), trained with Adam for 100 epochs; privacy experiments use 4 parties.
    • Number of repetitions: not reported in the paper.

    Main results

    Benign-setting accuracy (Table IV) and robustness (Table VI, robustness = accuracy under attack/benign accuracy):

    Swipe horizontally to read the table

    DatasetSettingMeanMedianMwuAvgMwuOptBulyanKrumCronus
    SVHNBenign accuracy95.994.893.994.494.589.691.1
    SVHNRobustness0.010.150.010.010.160.180.99
    MNISTBenign accuracy96.796.597.297.496.993.395.2
    MNISTRobustness0.090.950.260.130.970.960.99
    PurchaseBenign accuracy93.393.093.692.592.872.189.6
    PurchaseRobustness0.010.750.020.010.870.690.98
    CIFAR10Benign accuracy88.489.186.287.689.084.580.1
    CIFAR10Robustness0.130.170.160.150.850.210.97
    • The authors state that each existing aggregator is broken by at least one attack: Mean is affected by all attacks, Median by PAF, Bulyan/Krum by LIE, MwuAvg/MwuOpt by this paper's OFOM; on some datasets accuracy drops to near random guessing.
    • Cronus's strongest attack is mostly Label flip (LIE for CIFAR10); the authors report the accuracy drop under the strongest attack: Purchase 0.4%, SVHN 1.3%, MNIST 1.5%, CIFAR10 4.8% (main text; the abstract gives another set of numbers: 1.6%/1.3%/1.5%/2.1%).
    • Exception: LIE is ineffective against MNIST, which the authors attribute to the task being simple enough that the model can tolerate small perturbations; Bulyan is the most robust apart from Cronus, but its breaking point is low (it tolerates only 25–33% malicious parties).

    Membership inference (Table VII, 4 parties)

    • Passive attack: on Purchase, FedAvg single-party/aggregated-update inference accuracy is 77.1%/74.7%, and Cronus is 54.6%/53.6% (the main text gives another set of numbers: 78.1%/80.1% versus 51.7%/51.9%); SVHN and CIFAR10 show the same trend.
    • Active attack: on Purchase, FedAvg risk rises to 84.9%/82.7% (+7.8/+8.0 percentage points), while Cronus is only 54.9%/54.7% (+0.3/+1.1 percentage points).
    • Adversarial regularization (λ=3, Purchase): FedAvg risk drops by 9.9/12.7 percentage points, while Cronus drops by only a further 0.6/1.1 percentage points, which the authors attribute to Cronus's risk already being close to random guessing.

    Differential privacy and heterogeneous collaboration

    • User-level DP (SVHN, 32 parties): the authors state that even with ǫ=100, FedAvg's global model accuracy is close to random guessing, and Cronus gives similar results when there are few participants.
    • Record-level DP (Table VIII, SVHN, ǫ=15.4): with 32 parties, FedAvg benign accuracy is 65.7 and robustness 0.07, while Cronus is 85.8 and 0.97; with 4 parties, FedAvg is 81.6 and 0.02, while Cronus is 81.5 and 0.98; the authors state that under DP-SGD neither reaches stand-alone accuracy (87%), and DP-SGD FedAvg is not resistant to poisoning.
    • Heterogeneous collaboration (Purchase, 5 FC architectures A1–A5): 16-party heterogeneous collaboration has an average accuracy of 89.3, comparable to each homogeneous collaboration (Table IX); after adding 4 bad models (A1) to 12 good models, the good models' accuracy is essentially unchanged and the bad model reaches 78.1 (Table X); heterogeneity raises A1 from 70.1 to 78.1, and A2/A3/A4 rise by 3.2/2.2/1.4 percentage points respectively (Table XI).

    Other ablations and analyses

    • Sample complexity (Table II): the ratio of federated learning to Cronus is SVHN 1.2×10^5, MNIST 3.3×10^5, Purchase 2.75×10^5, CIFAR10 10.4×10^5.
    • Communication (Figure 3): randomly subsampling the public data each round at 2k/4k/6k/8k/10k gives final accuracies of 88.4/89.4/88.7/89.3/89.6, and the authors state that subsampling does not reduce the convergence speed.
    • Convergence (Figures 1, 4): under attack, Cronus's curves are almost identical to the benign setting; under benign settings, Cronus's accuracy is slightly lower than existing algorithms but higher than stand-alone.
    • The full attack matrix is in Appendix Table XII.
  5. What are some points that can be further explored?

    The authors note the DP utility and benign accuracy gap problems; the experiments are limited to 4 datasets and at most 32 parties.

    Limitations and future directions noted by the authors

    • Benign accuracy cost: The authors state that under the benign setting Cronus has a slight accuracy decrease compared with existing algorithms (Appendix B, Figure 4; e.g., 80.1% vs. FedAvg's 88.4% on CIFAR10, Table IV).
    • User-level DP is unsuitable for few participating parties: The authors state that UDP-FedAvg of McMahan et al. [14] cannot train a good model when there are few participating parties per round, and results on Cronus are similar (Section VII-C).
    • Insufficient DP utility: The authors state that existing DP algorithms (record-level [4], user-level [14]) have a large accuracy cost and are even lower than stand-alone training, and that DP algorithms with high utility and tight privacy loss need to be designed (Section VII-C).
    • Collaboration is ineffective under DP: The authors state that under DP-SGD the accuracy of both federated learning and Cronus fails to reach stand-alone training, making collaboration meaningless (Section VII-C2).
    • The paper has no dedicated Limitations section; the above are shortcomings explicitly stated in Results/Discussion.

    Scope of experiments

    • The evaluated models are 4 architectures (CNN, FC, FC, DenseNet-100), corresponding to 4 datasets in total: SVHN, MNIST, Purchase, CIFAR10; the number of benign parties is 16–32 (Table III).
    • The poisoning evaluation covers 4 attacks (Label flip, PAF, LIE, OFOM) × 7 aggregation configurations (Table XII); the membership inference evaluation uses Purchase, SVHN, CIFAR10 with 4-party collaboration (Section VII-B).
    • The heterogeneity experiment is conducted only on Purchase with 5 fully connected architectures (A1–A5) (Section VII-D).
    • The paper does not report the number of experimental repetitions or variance; the sample size of the membership inference experiments (the number of members/non-members per party) is not given as specific numbers in the main text.
    • The threat model assumes the attacker possesses data with the same distribution as the benign parties and knows 50% of the target party's members (Section II); the paper states this is a strong attacker assumption.
  6. Summarize the main content of the paper

    Cronus replaces parameter sharing with prediction sharing; the authors state that it achieves collaborative learning that is poisoning-resistant, low-leakage, and supports heterogeneity.

    A defense paper on secure collaborative learning that proposes the Cronus framework to address the poisoning vulnerability, privacy leakage and architectural homogeneity constraints caused by parameter sharing in federated learning.

    • Problem: the authors consider parameter sharing the most naive way of exchanging information in collaborative learning: it fully exposes the model's internal state to inference attacks, high-dimensional parameters make the error bounds and sample complexity of robust aggregation too high, and parameter-overwriting updates amplify the harm of poisoning.
    • Method: each party first trains independently on its local data, after which they share only the soft labels of their local models on unlabeled public data; the server aggregates predictions sample by sample using the dimension-independent robust mean algorithm of Diakonikolas et al. [17]; each party fine-tunes with its private data plus the soft-labeled public data, without overwriting parameters. The dimension of the exchanged vectors drops from the parameter scale to the number of classes.
    • Robustness: on SVHN, MNIST, Purchase and CIFAR10, the authors state that all existing federated aggregators (Mean, Median, MwuAvg, MwuOpt, Bulyan, Krum) are broken by at least one poisoning attack (including the OFOM proposed in this paper), whereas Cronus's robustness is 0.97–0.99, with an accuracy drop of 0.4%–4.8% under the strongest attack (Table VI).
    • Privacy: with 4-party collaboration on Purchase, passive membership inference accuracy against FedAvg updates is 77.1%, versus 54.6% for Cronus (close to random guessing at 50%); active attacks are almost ineffective against Cronus (Table VII).
    • Heterogeneity and efficiency: on Purchase, 5 different architectures can collaborate with accuracy no worse than homogeneous collaboration (Tables IX–XI); sample complexity is about 5 orders of magnitude lower than federated learning (Table II); subsampling the public data to 2k does not affect final accuracy (Figure 3).
    • Authors' conclusion: black-box knowledge transfer can simultaneously alleviate the three major limitations of federated learning—robustness, privacy and heterogeneity; the authors also note that benign accuracy decreases slightly and that existing DP algorithms have insufficient utility, requiring better high-utility DP designs.

Related papers

Show 3 more papers
Read originalarxiv.org