Privacy backdoor: stealing data with a tampered pretrained model

1. 01What problem does this paper attempt to solve?

   The authors propose a privacy backdoor: tampering with pretrained model weights suffices to steal fine-tuning data, reconstructing samples in the white-box setting and achieving perfect membership inference in the black-box setting while threatening DP-SGD.

   Core problem: the common practice of downloading and finetuning pretrained models from open-source repositories introduces an overlooked supply-chain privacy risk—a malicious model provider can steal finetuning data by tampering with weights.

   • Scenario and importance: Platforms such as Huggingface host nearly 500,000 models, and downloading and then finetuning is common practice. The authors state that prior supply-chain attack research focused on model integrity, and the privacy dimension has been overlooked.
   • Shortcomings of existing methods: Data-stealing attacks in federated learning (Fowl et al., 2021; Boenisch et al., 2023) assume the attacker can see individual gradient updates and only needs to survive one training step; the attacker in this paper can only see the final model, and the data trap needs to survive the entire finetuning process (multiple epochs, thousands of updates).
   • Core observation: The authors propose a single-use data trap design—the backdoor activates, writes the data point into the weights, and then becomes inactive, holding like a latch until the end of training, thereby capturing a single training sample with high probability and having little impact on model utility.
   • The paper proposes: privacy backdoor constructions for MLPs and pretrained transformers (ViT, BERT); perfect membership inference and data extraction under black-box; a tight end-to-end privacy attack on DP-SGD, which the authors state is the first tight end-to-end attack on this algorithm.
   • Threat model:
     • Goal: reconstruct a single finetuning sample of the victim, or infer whether a data point was used for training.
     • Knowledge: the attacker tampers with the weights of the pretrained model once before sending it to the victim; cannot observe individual gradient updates.
     • Capability: a white-box attacker can view the weights of the finetuned model; a black-box attacker can only make arbitrary input queries to the model.
     • Victim system: the victim performs full finetuning with SGD on a classification task (multiple epochs), adds a new linear classification layer to the model, and finetunes the entire model.

   Collapse this answer

2. 02What related research exists?

   Related work covers privacy attacks, integrity backdoors, federated learning data stealing and tightness analysis of DP-SGD; the authors state that this paper extends federated learning attacks to the pretrained model supply chain setting.

   Privacy attacks

   • Membership inference by Shokri et al. (2017) and training data extraction by Carlini et al. (2021; 2023) exploit the memorization effect of neural networks; Tramèr et al. (2022) and Panda et al. (2023) enhance memorization through poisoning.
   • Song et al. (2017) show that tampering with training code can make a model leak training data.

   Backdoor attacks (integrity direction)

   • Gu et al. (2017) and Liu et al. (2018) tamper with pretrained models to inject malicious target behavior, compromising model integrity.
   • Hong et al. (2022) show that such backdoors can be manually constructed by directly editing the weights of a pretrained model.

   Data stealing in federated learning

   • A malicious server tampers with the model so that client gradients leak training data (Zhao et al., 2023; Fowl et al., 2021; 2022; Boenisch et al., 2023; Wen et al., 2022; Phong et al., 2018).
   • The authors state that this paper extends the attack of Fowl et al. (2021), which creates sparse activations in linear layers so that gradients encode training inputs, enabling it to survive the entire finetuning process.

   Privacy backdoors in pretrained models (parallel work)

   • Wen et al. (2024) and Liu et al. (2024) concurrently proposed privacy backdoors in pretrained models, but the authors state that the two have weaker attacker goals (membership inference and extraction without success guarantees, respectively) and that the attacker has only black-box access; the authors consider these attacks easier to launch but less damaging, and not directly comparable to this paper.

   Tightness of DP-SGD

   • The worst-case analysis of DP-SGD by Abadi et al. (2016) assumes the attacker sees all training steps; Nasr et al. (2021) state that this analysis is tight for strong attackers who can see intermediate steps, but is considered loose for end-to-end attackers who only see the final model, and in practice privacy parameters are often set optimistically.

   Positioning: The authors state that this paper shifts the focus from integrity to privacy, that the attacker is weaker than in federated learning scenarios (seeing only the final model), and that it gives a tight end-to-end attack on DP-SGD; the paper does not set up comparison baseline methods in the traditional sense, and the experiments directly verify attack effectiveness on CIFAR-10/100, Oxford-IIIT Pet, Caltech 101, and TREC.

   Collapse this answer

3. 03How does the paper address this problem?

   The authors design a single-trigger data trap that writes into the weights and then self-locks, and for transformers design feature partitioning, sequence/position selectors and numerical stabilization techniques.

   Overall idea: implant a data trap in the pretrained model weights—a single-trigger backdoor unit that, when activated, writes the current training sample into the weights and then immediately shuts down, so that the captured input is not overwritten by subsequent updates throughout the fine-tuning process.

   Problem Setting and Single-Trigger Condition

   • For a linear unit h = ReLU(w^⊤ x + b), w and b are set so that with high probability only one input in a training batch activates it; after backpropagation the weight gradient is proportional to that input x, and dividing by the bias gradient recovers x.
   • Sufficient condition for a single trigger (shutdown): x̂⊤x + 1 > 0, and ∂L/∂h is positive and sufficiently large. The authors connect the backdoor output to the logits through a large weight w1⁽¹⁾, so that the captured input is misclassified with probability 1 − 1/C, thereby producing a large positive gradient that shuts down the backdoor; if it happens to be classified correctly (probability 1/C), the backdoor does not shut down and is repeatedly activated in subsequent batches.
   • Multiple backdoors: replicate this construction, with the bias set according to the activation proportion p such that p > 1/D and p ≪ 1/B (D is the dataset size, B is the batch size); k well-calibrated backdoors are expected to capture k·(1 − 1/C) inputs.

   Overall Architecture of the Transformer Backdoor

   • The 768-dimensional feature vector is logically partitioned into [ft, key, act]: ft carries benign features to preserve utility, key stores the information to be captured, and act propagates and amplifies the backdoor activation signal. The specific partitions are given in Appendix Table 2 (e.g., ViT full image: ft 448, key 256, act 64).
   • Module pipeline (Figure 3): input module constructs feature separation → backdoor module (the MLP of the first encoder block) captures the input → amplifier module amplifies the activation → erasure module clears the key features → n signal propagation modules pass act through shortcuts → output module averages act into the CLS token via attention at the last layer.

   Capturing the Full Input: keyed backdoor

   • key is further split into [tok, pos, seq]: tok is the token information to be captured (for ViT, a downsampled grayscaled 32×32 patch; for BERT, a word embedding subvector); pos is a near-orthogonal position identifier; seq is a sequence identifier, obtained by averaging over all tokens in the first attention layer.
   • The backdoor weights take the form w⁽ⁱ⁾ = [0, pos⁽ⁱ⁾, w_seq], with the bias set to a high quantile, so that the backdoor activates only for tokens at a specific position in a specific sequence; multiple backdoor families replicate this design to capture multiple full inputs.

   Numerical Stabilization Techniques

   • Layer normalization: by adding a large constant C to the feature components, the StabLN module "virtually splits" the normalization, eliminating input-dependent scaling and coupling between components (Appendix A.1).
   • GELU: it cannot be fully shut down like ReLU, so the authors amplify the input to make it approximate ReLU, and use a subsequent MLP to suppress the unintended gradients corresponding to small activations (Appendix A.2).
   • Shutdown enhancement: SpecLinear (an LN+MLP combination) is used to increase the magnitude of the shutdown term without affecting selectivity (Appendix A.3).

   Black-Box Attack and DP-SGD Application

   • Membership inference: the penultimate layer activations are set so that the target x produces h = [C, 0, …, 0] and other inputs h₁ = 0; the attacker compares the change in logits before and after fine-tuning to determine whether it participated in training.
   • Data extraction: the problem is reduced to model stealing, finding the critical point c = −b/w₀ of each ReLU backdoor unit by querying x = [c, 0, …, 0], recovering the weight w coordinate by coordinate, and thereby reading out the captured input.
   • Success determination: images are judged as a unique match by the backdoor activation strength threshold during training (Appendix D.1); for text, all training sentences that strongly activate the backdoor are reported.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Under white-box, recognizable images and near-complete sentences are reconstructed from MLP, ViT, and BERT; black-box membership inference is 100% accurate, and the DP-SGD lower bound reaches 0.72–0.73 of the theoretical upper bound.

   Experimental Setup

   • Models tested: 3-layer MLP (256 hidden units, of which 64 are used as backdoors); ViT-B/32 (Torchvision pretrained); BERT-base (HuggingFace); for the DP experiments, a small CNN pretrained on CIFAR-100 (convolutional layers + 3-layer perceptron, hidden layers 128, 64).
   • Datasets: CIFAR-10/CIFAR-100 (50,000 training samples), Oxford-IIIT Pet (3,680 training samples), Caltech 101 (9,146 images, two-thirds used as the training set), TREC (5,452 training questions, 6 coarse classes/50 fine classes).
   • Training configuration: MLP uses SGD, batch 64, lr 0.05, 20 epochs; ViT uses lr (0.2, 1e-4), batch 128, 12 epochs; BERT uses lr (0.05, 1e-4), batch 32, 12 epochs; the backdoor threshold is set by quantile Q(0.001), with about 10 samples activating each backdoor.
   • Metrics and criteria: reconstruction quality is presented as a match with ground truth (images are judged as a unique match by an activation strength threshold; for text all strongly activated sentences are listed); membership inference accuracy; the DP experiments use the ratio of the numerically computed privacy lower bound ε̃ to the theoretical upper bound ε; downstream test accuracy.
   • Review method: no LLM or human review; match determination is based on an activation strength threshold (a hyperparameter, manually set by the authors).
   • Number of repetitions: the paper does not report the number of repetitions for each experiment; Table 7 is labeled as the "cleanest" run (best run) among multiple trials.

   Main Results

   The paper mainly presents reconstructed example figures/tables, with no quantitative main results table; the key quantitative results are as follows (from the main text and appendix):

   Setting	Result	Source
   MLP + CIFAR-10, white-box reconstruction	Most reconstructions are hard to distinguish from ground truth; two failure types: unrecoverable due to misclassification, and mixing of multiple inputs	Figure 2
   ViT-B/32 + Caltech 101, white-box full-image reconstruction	Grayscale reconstructions are recognizable, with slight deviations in patch brightness	Figure 4
   BERT + TREC-50, white-box sentence reconstruction	Reconstructions are near-perfect, with occasional spurious suffix tokens	Table 1
   backdoored ViT downstream accuracy	83% test accuracy (Caltech 101)	Section 5.4
   backdoored BERT downstream accuracy	78% test accuracy (TREC-50)	Section 5.4
   Black-box membership inference	100% accuracy	Section 6.1

   • The authors state that in the transformer experiments the number of backdoors is smaller than the number of classes, and all backdoors shut down with high probability; what is reported are fine-tuning runs in which the model's benign utility did not collapse.
   • The authors state that the ReLU version of the transformer's backdoors is more robust and captures more successfully (Appendix F).

   DP-SGD Tightness Experiments

   • What was measured: a backdoored CNN (frozen backbone) fine-tuned with DP-SGD (C = 1, σ = 1.0, δ = 1e-5, lot size 500), with the attacker seeing only the final model, and the privacy lower bound ε̃ computed numerically.
   • Results: at gradient concentration ρ = 0.97, ε̃/ε is 0.64 (ε=1), 0.73 (ε=3), 0.72 (ε=5), 0.72 (ε=8) respectively (fine-tuning only the classification head); when fine-tuning the entire MLP, at ε=3 it is 0.72 (Table 4). Figure 5 shows the empirical privacy budget approaching the theoretical upper bound.
   • Authors' interpretation: the authors state that the loose privacy budgets used in the literature (e.g., ε > 8) are unsafe when the model provider is untrusted, challenging the common assumption that "DP-SGD guarantees are too conservative in practice."

   Black-Box Data Reconstruction

   • What was measured: for the backdoored MLP on CIFAR-10, partial model stealing is used to recover the first-layer backdoor weights coordinate by coordinate.
   • Results: the model output approximates ReLU, and two queries suffice to find the critical point; the full attack requires fewer than 10,000 queries and recovers recognizable CIFAR-10 images (Figure 6).
   • Authors' interpretation: the authors state that this problem is simpler than general model stealing (only the first layer is needed, high precision is not needed, and the backdoor units are connected with large weights).

   Utility Cost and Ablations

   • ViT utility (Table 3, Caltech 101): full model test 95.0% (GELU)/95.1% (ReLU); after pruning, the benign submodel 91.3%/91.9%; after adding the full backdoor it drops to 82.6% (GELU)/67.8% (ReLU). The authors state that under GELU the roughly ten percentage point drop can be compensated by backdooring a larger model.
   • crafted vs. random classification head (Figure 13): with a crafted head there are no unrecoverable backdoors even with few classes; with a random head, CIFAR-100 has fewer unrecoverable backdoors than CIFAR-10.
   • Patch-level reconstruction (Figure 15): on Oxford-IIIT Pet some reconstructed patches contain recognizable information (e.g., a dog's face), while others are unrecognizable.
   • Sensitivity of the DP lower bound to ρ (Figure 12): a strong lower bound is still obtained at ρ = 0.90.
   • Negative results: under GELU the backdoor cannot be fully shut down and persistently captures a small fraction of inputs, reducing reconstruction quality; large gradients from failed backdoors may damage model utility, and the authors state that careful tuning is required (Appendix C.4).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that finetuning methods such as LoRA and backdoor robustness remain to be studied; the experiments cover 4 models and 5 datasets, and the number of repetitions is not reported.

   Limitations noted by the authors and future directions

   • Finetuning methods: The authors state that extending to other finetuning types (such as LoRA) is left as future work (Section 3, Section 5).
   • Robustness challenges: The authors state that ensuring the robustness of the backdoor construction and model utility requires more careful tuning, and is the main challenge when extending to large transformers (Section 4.1, 5.1, Appendix C.4).
   • Utility drop: The authors state that the backdoor reduces downstream accuracy, by about ten percentage points under GELU, which can be compensated by backdooring a larger model or more careful tuning (Section 5.4, Appendix D.2).
   • Threshold estimation: The attacker cannot access the real private finetuning dataset and needs to estimate the quantile threshold using a similar public dataset (Appendix D.3).
   • DP experimental setup: The authors state that freezing some weights simplifies the backdoor design and analysis; when the victim randomly initializes the classification head, the stability of the backdoor relies on a heuristic assumption, and in the experiments this failure probability is ignored based on a rough calculation (Section 6.1, Appendix E.2).

   Scope of experiments

   • 4 models tested in total: 3-layer MLP, ViT-B/32, BERT-base, CIFAR-100 pretrained small CNN (Section 4.3, 5.4, Appendix E.3).
   • 5 datasets in total: CIFAR-10, CIFAR-100, Oxford-IIIT Pet, Caltech 101, TREC (Appendix D).
   • White-box reconstruction covers images (full-image and patch-level) and text; black-box experiments cover membership inference and data extraction on MLP, and black-box data extraction is not demonstrated on transformers (Section 6.2).
   • DP experiments are conducted on CIFAR-10 with σ = 1.0, δ = 1e-5, covering ε ∈ {1, 3, 5, 8} (classification head only) and ε = 3 (MLP) (Appendix E.4).
   • The paper does not report the number of repetitions or statistical significance tests for each experiment; Table 7 notes that it is the best single run among multiple trials.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors propose a privacy backdoor attack: tampering with the weights of a pretrained model is enough to steal finetuning data, with white-box sample reconstruction, perfect black-box membership inference, and bringing the DP-SGD lower bound close to the upper bound.

   A supply chain privacy attack paper: a malicious model provider tampers with pretrained model weights before release, and after the victim fine-tunes, the attacker recovers the fine-tuning data from the model.

   • Problem: downloading a pretrained model and then fine-tuning is common practice, and the authors state that prior supply chain attack research focused on integrity, with privacy risk overlooked; federated learning data stealing attacks assume the attacker can see gradients, whereas the attacker in this paper can only see the final model.
   • Method: the core is a single-use data trap—when a backdoor unit activates, it writes the training sample into the weights and then immediately shuts down, so the data survives the entire fine-tuning process. On MLPs it is constructed based on linear layers; on transformers, the input is captured through a feature three-way partition [ft, key, act], sequence/position selectors, and numerical techniques such as StabLN and GELU handling to ensure signal propagation.
   • White-box results: most images reconstructed by the MLP on CIFAR-10 are hard to distinguish from ground truth (Figure 2); ViT-B/32 reconstructs recognizable full images on Caltech 101 (Figure 4); BERT reconstructs near-perfect sentences on TREC-50 (Table 1); the backdoored ViT and BERT maintain 83% and 78% test accuracy respectively.
   • Black-box results: membership inference reaches 100% accuracy; data extraction on the MLP requires fewer than 10,000 queries.
   • DP-SGD results: on the backdoored CNN, the empirical privacy lower bound of the end-to-end attack approaches the theoretical upper bound, with ε̃/ε up to 0.73 (Table 4); the authors state that loose privacy budgets (e.g., ε ≥ 8) are unsafe when the model is untrusted.
   • Authors' conclusion: model backdoors break not only integrity but also privacy, and worst-case weights must be considered when reasoning about the privacy of deployed models; the authors plan to release reproduction code. Limitations include covering only full finetuning, the need for careful tuning of transformer backdoor robustness, and the backdoor reducing downstream accuracy.

   Collapse this answer

Related papers