ShadowFPT: Backdooring Federated Prompt Tuning via Shadow Triggers

When federated prompt tuning aggregates only lightweight prompts and the backbone is frozen, can a malicious client implant a targeted backdoor while keeping uploaded updates close to benign updates?

• Setting: Federated Prompt Tuning (FPT) freezes the vision–language backbone, clients update prompts with private data, and the server receives only prompts. The authors argue this reduces communication and parameter overhead, but the shared and frozen backbone becomes an overlooked attack surface.
• Existing approaches: Classic backdoors and federated backdoors mostly rely on data poisoning or anomalous model updates. Federated defenses mostly assume that malicious behavior will appear in uploaded updates or in representations derived from them. The authors argue this assumption becomes fragile in FPT: the server sees only low-dimensional prompts, and the representation geometry is determined by the shared frozen backbone.
• This paper's approach: Proposes ShadowFPT. It shifts the attack burden from conspicuous prompt poisoning to guiding the frozen representation with a trigger: first pretrain a Shadow Trigger, then adapt the trigger during federated training, and optimize local prompts with clean and triggered samples, uploading only prompts.
• Threat model:
  • Goal: Maintain the global model's clean accuracy and cause triggered inputs to be classified as the attacker-specified class yt. Triggered inputs are denoted xδ=A(x, δ).
  • Knowledge: white-box. The malicious client knows the structure of the frozen backbone.
  • Capability: Has a local target-class sample set Ryt. Locally it can jointly optimize the prompt and the trigger δ, but uploads only the prompt. It cannot modify the frozen image encoder, the frozen text encoder, the global training protocol, or the server aggregation rule.
  • Victim system: FPT on a frozen vision–language backbone. The server aggregates client-uploaded prompts with the aggregation rule Agg.

Federated prompt tuning

• CLIP (Radford et al., 2021): learns transferable image–text representations and is treated by the authors as a foundation model for multimodal adaptation.
• Prompt tuning: text prompts (Zhou et al., 2022a,b), visual prompts (Jia et al., 2022), joint vision–language prompts (Khattak et al., 2023a), and semantic regularization (Khattak et al., 2023b). The common approach is to freeze the pretrained backbone and optimize only a small number of prompt parameters.
• Federated prompt learning: cooperative prompt learning (Guo et al., 2023b), personalization (Guo et al., 2023a), text-driven prompt generation (Qiu et al., 2024), the trade-off between personalization and generalization (Cui et al., 2024), mixture-of-experts prompts (Luo et al., 2025), and multimodal visual prompt tuning (Singha et al., 2025). The authors state that these works mainly concern communication efficiency, personalization, and generalization on heterogeneous data, and that the backdoor risk of FPT remains insufficiently understood.

Backdoor attacks

• BadNets (Gu et al., 2017) and Blended (Chen et al., 2017): poison training data with visible or blended triggers so that the model keeps its original behavior on clean inputs and predicts the specified label on triggered inputs.
• Federated backdoors: stealthy update poisoning, persistent injection, adaptive optimization, colluding triggers, and personalized federated learning; the paper names Lyu et al. (2023), Zhang et al. (2022), A3FL (Zhang et al., 2023), Lyu et al. (2024a), and Bad-PFL (Fan et al., 2025).
• BadCLIP (Liang et al., 2024): manipulates multimodal contrastive representations using visual and text embedding guidance. The authors state that existing attacks mostly revolve around data poisoning, model update manipulation, or directly poisoning prompt parameters; the parameters that FPT can train and that are communicated are very small, so the capacity to inject a backdoor through uploaded updates is limited.

Backdoor defenses

• Byzantine-robust aggregation: Krum (Blanchard et al., 2017), Trimmed-Mean (Yin et al., 2018), Bulyan (Guerraoui et al., 2018); later work uses geometric aggregation, trust guidance, or sparsification (Pillutla et al., 2022; Karimireddy et al., 2022; Cao et al., 2020; Xu et al., 2025).
• Against federated backdoors: FLAME combines client clustering with model clipping (Nguyen et al., 2022); CRFL provides certified robustness against federated backdoors (Xie et al., 2021). There are also secure aggregation, robustness under privacy constraints in heterogeneous settings, system- and optimization-level mitigation, and identification based on dynamic behavior (Huang et al., 2025; Zhang et al., 2025).
• SABRE-FL (Khan et al., 2025): performs anomaly analysis in embedding space to detect malicious clients. The authors state that the above defenses, despite different methods, all assume that malicious behavior will manifest as detectable anomalies in uploaded updates or in representations derived from them; under non-IID settings benign prompt updates themselves can differ greatly, and this assumption becomes fragile in FPT.

Baselines and datasets

• Attack baselines: BadNets, Blended, Neurotoxin, A3FL, FCBA, CerP, Bad-PFL, PFedBA.
• Aggregation and defense baselines: FedAvg, Krum, TrimmedMean, Bucketing, LaSA, FoolsGold, FLTrust, FLAME, CRFL.
• Datasets: Caltech101, Oxford Flowers, UCF101.

The authors position this paper as: exploiting the shared frozen CLIP representation space to shift the attack from anomalies in prompt updates to trigger-induced representation guidance.

ShadowFPT has two stages: first learn a shadow trigger on the frozen CLIP representation space, then implant a targeted backdoor during local prompt training on the malicious client. The trigger and frozen encoders stay local; the server only sees the prompt.

Problem setting

• The system contains an image encoder fI and a text encoder fT, both frozen throughout. N clients each have private data Di, and the distributions may be non-IID. The label set is C. The main text first writes the formulation for text prompt tuning; experiments additionally test visual and joint prompts.
• The class logit is the cosine similarity between the image feature and the text feature of that class's prompt, yielding the prompt classifier F(x; P). In round t, the server sends the global prompt P^t to a client subset St; clients optimize prompts on local data with cross-entropy, then Agg aggregates. In experiments all clients participate in every round.
• The attacker wants triggered inputs to be predicted as yt while maintaining clean accuracy. The malicious client has a local target-class image set Ryt.

Shadow trigger pretraining

• The pretraining data Ds can be auxiliary public data or the malicious client's local data. Only the trigger δ is updated; both encoders are fixed. The triggered image is denoted xδ=A(x, δ).
• Target text alignment: this stage uses fixed handcrafted CLIP templates, not learnable federated prompts. The paper's example template is "a photo of a [CLASS]". The text anchor for each class is the average of frozen text features over the template set; the loss pulls the final image feature of the triggered image toward the target-class anchor zt.
• Layer-wise visual alignment: on a selected set of visual layers K, target-class prototypes are computed using Ryt, and the intermediate features of the triggered image are pulled toward these prototypes. The authors write that this term is intended to constrain how the trigger changes intermediate visual representations, not only the final image-text similarity.
• The pretraining objective is L_shadow=L_text+λ_l L_layer. It represents a weighted sum of final semantic alignment and layer-wise visual alignment. The resulting trigger is denoted δshadow. The paper does not give specific values for λl and K.

Injection during federated training

• In round t, the malicious client receives P^t, initializes the local prompt as Pa←P^t, then performs two steps, and the order of the two steps is fixed.
• Trigger adaptation: the first attack round initializes with δshadow, and subsequent rounds continue updating the trigger locally. This step fixes the prompt to the current P^t and optimizes only the trigger. The target text anchor is changed to fT(yt; P^t) under the current global prompt; the layer-wise term still uses the target-class visual prototypes defined in Section 3.3. This produces the trigger δ^(t) for that round.
• Prompt optimization: with δ^(t) fixed, Pa is optimized on the malicious client's local data. The objective is L_back(P_a)=E[CE(F(x;P_a),y)]+E_{y≠ yt}[CE(F(x_δ;P_a),y_t)]. The first term preserves the clean task, and the second term pulls triggered versions of non-target samples toward yt.
• The overview says the trigger acts on a subset of the local data; the loss formula takes the expectation over all local samples with y≠yt. The paper does not give the subset proportion. After optimization only Pa is uploaded; δ^(t) and the frozen encoders are not communicated. The server still uses the same Agg as standard FPT.

Variants and implementation choices given in the paper

• ShadowFPT-I: the shadow trigger is pretrained using only the malicious client's local data. ShadowFPT-II: uses auxiliary public data, instantiated as 100 randomly drawn ImageNet classes. The paper states that it does not assume these classes have semantic overlap with the downstream target classes.
• Both variants use a 32×32 trigger patch located at the bottom-right corner of the image. The trigger pretraining learning rate is 0.01, and the federated adaptation learning rate is 0.001. The local prompt optimization learning rate is 0.0035.
• The success criteria are defined in experiments as: clean accuracy Acc on benign test samples, and attack success rate ASR on triggered non-target test samples. The paper does not write a separate evaluation model or discrimination threshold.

Experimental setup

• Backbone and prompts: All methods use frozen CLIP with a ViT-B/16 backbone. The default is text prompt tuning, FPT-L; visual prompt FPT-V and joint vision–language prompt FPT-VL are also tested.
• Federated split: Following the FedTPG setting, 20 clients; each client has 5 classes, and class sets are disjoint across clients. All clients participate in each round. The default is 1 malicious client, and the target class is the first class of each dataset. 20 communication rounds, 1 local epoch per round, batch size 32, prompt learning rate 0.0035.
• Data: Caltech101, Oxford Flowers, UCF101, corresponding to general objects, fine-grained flowers, and human actions, respectively. The paper does not give the sample size of each dataset. The auxiliary data for II is 100 random ImageNet classes.
• Attack baselines: BadNets, Blended; Neurotoxin, A3FL, FCBA, CerP; Bad-PFL, PFedBA.
• Aggregation and defenses: FedAvg; Krum, TrimmedMean, Bucketing; LaSA, FoolsGold; FLTrust; FLAME, CRFL. SABRE-FL from the introduction does not appear in this experimental list.
• Metrics: Acc on benign test samples; ASR on triggered non-target test samples. Table 1 averages over all aggregation and defense rules; Table 2 averages over all datasets and prompt settings. The paper does not report repeated random seeds, error bars, or statistical tests, and does not use a separate judge model.

Main results

The table below lists only ShadowFPT-II; the numbers are averaged over all aggregation and defense rules (according to Table 1). Δ is the percentage-point difference relative to that row's Best Base. Best Base is the non-ShadowFPT attack with the highest average ASR on that dataset–setting. Not listed: Caltech101's FPT-V and FPT-VL, and UCF101's FPT-V. Among the nine rows, II's lowest ASR is the listed Caltech101, FPT-L.

Swipe horizontally to read the table

Setting (Best Base)	Best Base ASR	II Acc	II ASR	ΔAcc	ΔASR
Caltech101 FPT-L (A3FL)	19.81%	87.67%	90.36%	-0.82	+70.55
Oxford Flowers FPT-L (A3FL)	38.07%	60.85%	98.93%	-5.78	+60.86
Oxford Flowers FPT-V (PFedBA)	30.22%	66.61%	98.94%	+0.88	+68.72
Oxford Flowers FPT-VL (PFedBA)	54.60%	68.33%	99.18%	-1.56	+44.58
UCF101 FPT-L (PFedBA)	25.06%	65.87%	99.98%	-2.15	+74.92
UCF101 FPT-VL (Blended)	22.49%	67.89%	99.63%	-2.25	+77.14

• The authors state that after averaging over nine settings, Best Base has ASR 34.56%, ShadowFPT-I has 95.63%, and ShadowFPT-II has 97.67%; they also state that II is the method with the highest ASR in every setting. For the three rows not listed above, II's ASR in Table 1 is likewise higher than that row's Best Base.
• The authors state that the clean-accuracy drop is limited: on Caltech101, II drops by at most 1.05 points (corresponding to FPT-V, which is not listed above); on UCF101 the drop is 0.55 to 2.25 points, of which 0.55 comes from the unlisted FPT-V. The authors argue that the clean–attack trade-off is stronger on Oxford Flowers FPT-L, and much weaker on FPT-V and FPT-VL.
• The authors state that the comparison with Best Base is conservative, because the comparator is already the strongest non-ShadowFPT attack on each dataset–setting. The main-setting numbers in the abstract are consistent with Caltech101, FPT-L in Table 1: ASR from 19.81% to 90.36%.

Against aggregation and defense rules

• Table 2 averages over all datasets and the three prompt settings. The authors state that ShadowFPT-II has the highest ASR under every defense, ranging from 96.63% to 98.30%; the strongest non-ShadowFPT baseline ranges from 13.17% to 60.93%. In the table, II's lowest ASR is under Krum (96.63%) and its highest under FedAvg (98.30%); Best Base's lowest ASR is under FLAME (A3FL, 13.17%).
• ΔASR values named by the authors: Krum +73.97, LaSA +76.55, FLAME +84.75, CRFL +74.55 percentage points. The authors state that II's average ASR is 63.61 percentage points higher than Best Base, and clean accuracy drops by 1.71 points on average.
• In Table 2, II's largest ΔAcc drop is under CRFL, at -3.80; under FoolsGold, I's ΔAcc is +2.01. The authors argue that these defenses suppress anomalous updates through robust aggregation, filtering, clustering, clipping, or certification, whereas ShadowFPT places much of the backdoor effect in the interaction between the adapted trigger and the frozen representation; they also state that its robustness comes mainly not from harming benign performance.

Prompt modality and prompt space

• Figure 2 averages over the three datasets and all aggregation or defense rules. The main text gives ShadowFPT-II's ASR on FPT-L, FPT-V, and FPT-VL as 96.42%, 97.63%, and 98.95%; Best Base is 27.65%, 33.20%, and 42.83%. The authors state that ShadowFPT-I's ASR is above 93% on all three modalities. Relative to Best Base, II's average Acc drop is 2.92, 0.24, and 1.45 percentage points. The authors argue that the attack does not depend on a single prompt modality, but exploits the frozen CLIP representation space shared by the three settings.
• Figure 3 is a PCA of client prompts under FedAvg. The caption states: blue dots are benign clients, red crosses are malicious clients, and green stars are the benign centroid. The authors state that the malicious prompts of existing attacks are outliers relative to the benign cluster, whereas ShadowFPT is close to benign clients and almost overlaps with the benign centroid. The converted text cannot show the coordinates of the points; the positional judgment comes from the main text, not from values readable in the figure.

Pretraining, injection, and alignment

• Figure 4 compares components on Caltech101. Main text: the average ASR of a randomly initialized trigger is 21.32%; with only the pretrained shadow trigger and no federated injection it is 49.75%, of which Krum is 25.17% and FLTrust is 31.99%; the full method has an average ASR of 90.36%, which is +59.54 on Krum and +60.55 on FLTrust relative to no injection. The authors state that pretraining provides a transferable initialization, and federated injection is what turns it into a backdoor that can pass through defenses.
• Figure 5 compares alignment terms on Caltech101 under nine aggregation and defense rules. Target-text alignment alone has an average ASR of 66.38%, of which CRFL is 20.08%; layer-wise visual alignment alone is 59.25%, of which FoolsGold is 20.73%; the two together are 90.36%, 23.98 and 31.11 percentage points higher than the two single objectives. The authors state that text alignment provides the semantic target, and layer-wise alignment stabilizes the trigger representation trajectory.
• The above 90.36% is the same as the ASR of Caltech101, FPT-L, ShadowFPT-II in Table 1. The caption does not state whether it is Variant I or II, nor the prompt setting.

Other ablations and analyses

• Apart from Figure 4 and Figure 5, the paper does not report further sweeps of λl, the layer set K, the trigger patch size, or the trigger subset ratio.
• The experiments default to 1 malicious client; the main text says "unless otherwise stated" but does not give separate results for the number of malicious clients.
• Appendix Table 3 lists Acc and ASR for each attack under three datasets, three prompt settings, and nine defenses. After conversion, the difference rows are not fully consistent with the average ΔAcc in Table 1, so they are not cited cell by cell.

Limitations noted by the authors and future directions

• Item 2 of the NeurIPS Paper Checklist states: the current version has no dedicated limitations discussion. The limitations listed by the authors include the white-box threat model, focus on CLIP-style frozen backbones, the default of one malicious client, and evaluation only on three public image recognition datasets.
• Item 7 of the Checklist states: results are averaged over datasets, prompt settings, and aggregation or defense rules, but no error bars, confidence intervals, or statistical significance tests over repeated random seeds are reported; the authors state this is a limitation of the current empirical report.
• Item 5 of the Checklist: the current submission does not provide open-source code or reproduction scripts. The authors plan to release anonymous code and reproduction instructions when the submission policy permits.
• Item 8 of the Checklist states that GPU type, memory, per-experiment runtime, and total compute budget are not provided, and states that these details should be added to the appendix.
• Section 5: the authors believe protecting FPT requires going beyond prompt anomaly detection and considering the interaction between triggers and the geometry of frozen representations. This is a recommendation in the conclusion, not a completed defense.

Scope of experiments

• The backbone tested is frozen CLIP ViT-B/16; prompt settings are FPT-L, FPT-V, FPT-VL (Section 4.1).
• Evaluation datasets are Caltech101, Oxford Flowers, UCF101. Federated setup is 20 clients, a five-class partition with disjoint class sets, full participation each round, 20 rounds, and 1 local epoch per round (Section 4.1).
• Aggregation and defenses in experiments are FedAvg, Krum, TrimmedMean, Bucketing, LaSA, FoolsGold, FLTrust, FLAME, CRFL (Section 4.1–4.2, Table 2). The paper does not report experimental numbers for SABRE-FL from the introduction.
• Attack baselines are BadNets, Blended, Neurotoxin, A3FL, FCBA, CerP, Bad-PFL, PFedBA. ShadowFPT-II's auxiliary data are 100 random ImageNet classes, and the paper states it does not assume semantic overlap with downstream classes (Section 4.1).
• The default is one malicious client, and the target class is the first class of each dataset; metrics are benign test Acc and triggered non-target test ASR. The paper does not report λl, the visual layer set K, the trigger subset proportion, number of repetitions, or error bars (Section 3–4; checklist item 7).

ShadowFPT is a targeted backdoor against federated prompt tuning. FPT freezes the shared CLIP backbone and aggregates only prompts; the authors argue that the backdoor can come mainly from trigger-induced guidance of the frozen representation, while uploaded prompts can still be close to benign updates.

The threat setting is white-box: the malicious client knows the structure of the frozen backbone and can optimize the local prompt and the trigger, but cannot modify the encoders, the training protocol, or the aggregation rule, and uploads only prompts. The goal is to maintain clean accuracy and cause triggered inputs to be classified as the specified class.

The method has two stages. The shadow trigger is pretrained on the frozen vision encoder: the final embedding of triggered images is aligned to the target-class text anchor, and intermediate features are aligned to layer-wise prototypes obtained from local target-class samples. During federated rounds, the trigger is first adapted under the current global prompt, and then the local prompt is optimized with clean samples and triggered non-target samples. Variant I pretrains the trigger with local data; Variant II uses 100 random ImageNet classes, and the paper states that it does not assume semantic overlap with downstream classes.

• Table 1 main setting (Caltech101, FPT-L, averaged over all aggregation and defense rules): ShadowFPT-II has ASR 90.36% and Acc 87.67%; Best Base is A3FL, with ASR 19.81% and Acc 88.49%. The authors state that ASR increases by 70.55 percentage points.
• The authors state that after averaging over nine dataset–setting combinations, Best Base has ASR 34.56%, ShadowFPT-I has 95.63%, and II has 97.67%.
• In Table 2 (averaged over all datasets and prompt settings), II has ASR from 96.63% to 98.30% under nine defenses. The authors state that average ASR is 63.61 percentage points higher than Best Base, and Acc drops by 1.71 points on average.
• In Table 1, for Oxford Flowers, FPT-L, II has ΔAcc of -5.78, the largest clean-accuracy drop in that table. The authors argue that the clean–attack trade-off is stronger on fine-grained recognition.
• The authors state that in the PCA under FedAvg, its malicious prompts are close to benign clients and almost overlap with the benign centroid; they also argue that prompt anomaly detection alone is insufficient to protect FPT. The frozen backbone reduces trainable parameters, but the authors argue this does not eliminate the backdoor attack surface.