BAPLe++: Cross-Model Transferability of Backdoors in Medical Foundation Models and Prompt-Level Defense

Core question: does a backdoor prompt implanted into a Med-FM via prompt learning remain effective after transfer to another Med-FM? Can an infected prompt be repaired with the backbone frozen?

• Scenario and importance: Med-FM is entering clinical decision support workflows and is commonly adapted using public data and open-source checkpoints, which the authors believe brings attack surfaces such as data poisoning and checkpoint tampering; backdoored models perform normally on clean data and are hard to detect with routine validation, making them especially dangerous in clinical settings.
• Shortcomings of existing work: the authors point out that BAPLe only constructed the attack and did not evaluate transfer from the source model to the target model; defenses for Med-FM under few-shot prompt learning lack systematic study; defenses for natural images (denoising, activation clustering, adversarial training) are largely unexplored in the Med-FM setting with scarce labels and frozen backbones.
• Core observation of the paper: the backdoor resides in the learnable prompt rather than the model weights, so defense can be carried out entirely at the prompt level, treating the suspicious prompt as a reference to an already contaminated region in embedding space.
• What the paper proposes: first, using the new metrics CAD and BAD to quantify the cross-model transfer of backdoor prompts (the authors state this is the first study of this problem); second, proposing the defense method DePLe and comparing it with four training-free input transformations.
• Threat model:
  • Attacker goal: implant a targeted backdoor during downstream adaptation, with clean inputs behaving normally, triggered inputs classified into the attacker-specified class, and stealth maintained under standard validation based only on clean data.
  • Attacker knowledge and capability: can only poison a small amount of downstream training data, and can apply a trigger to inputs at inference time; because medical labels are scarce and compute is limited, full fine-tuning is infeasible, so prompt learning is used.
  • Attack vectors: two scenarios — a malicious service provider implants a backdoor into the task prompt returned to the user; or backdoored prompts are distributed via a public repository (e.g., Hugging Face).
  • Victim system and defender: an institution deploying Med-FM for downstream medical tasks; the defender holds the pretrained Med-FM, the possibly infected deployed prompt and a small clean labeled set, and cannot retrain or modify the backbone.

Medical foundation models (Med-FM)

• MedCLIP (Wang et al., 2022) — adapts the CLIP framework to chest X-rays, aligning image and report features for zero-shot classification.
• BioMedCLIP (Zhang et al., 2025) — trained on millions of medical image-text pairs, extending cross-modal transfer.
• PLIP (Huang et al., 2023) and QuiltNet (Ikezogwo et al., 2023) — pretrained models for pathology analysis at the cell and tissue level.

Prompt learning

• CoOp (Zhou et al., 2022) — learns tunable prompt tokens to adapt vision-language models such as CLIP; this paradigm was later transferred to the medical domain (MedCLIP, PLIP). The paper points out that the mechanism of frozen encoders plus parameter-efficient adaptation introduces new security risks.

Backdoor attacks

• BadNets (Gu et al., 2017), Trojaning (Liu et al., 2018), WaNet (Nguyen & Tran, 2021) — from patch triggers to more stealthy and more generalizable attacks.
• FIBA (Feng et al., 2022) — a backdoor for medical images; BadCLIP (Liang et al., 2024) — couples image triggers with prompt embeddings to attack vision-language models.
• BAPLe (Hanif et al., 2024) — the paper states it is the first work to inject backdoors into Med-FM under few-shot prompt learning, and this paper directly extends it.

Adversarial transfer and defense

• RAP (Qin et al., 2022) and GRA (Zhu et al., 2023) — improve the cross-architecture transfer of adversarial perturbations by reshaping the loss surface and gradient signals. The paper points out that in the backdoor setting what transfers is the learned prompt rather than the input perturbation, and this phenomenon has not yet been studied on Med-FM.
• Test-time preprocessing defenses (blur, compression, filtering) are easy to deploy, but their robustness/utility trade-off on Med-FM is unclear (per the paper).

Baselines and benchmarks: the attack baselines are BadNets, WaNet, FIBA (results taken from the BAPLe original source) and BAPLe itself; the defense baselines are four training-free input transformations: Gaussian Blur, JPEG-30, Feature Squeeze (MF-3), Pixel Deflection. The benchmarks are six datasets: COVID, RSNA18, MIMIC, Kather, PanNuke, DigestPath.

The authors position this paper as: building on the BAPLe attack, filling the two gaps of cross-model transfer evaluation and prompt-level defense.

The overall approach has two parts: first, BAPLe is used on the source model to create a backdoor prompt and trigger, which are transferred as-is to the target model and the transfer strength is measured with CAD/BAD; then DePLe is proposed to repair the infected prompt in prompt space, with the backbone frozen throughout.

Problem setting and attack (BAPLe)

• The Med-FM is a CLIP-style dual encoder fθ = {fI, fT}, and zero-shot classification takes the argmax over the cosine similarity between image features and the features of each class's text template.
• Prompt learning replaces the handcrafted template with learnable token embeddings p, with the class text being ti = [p; yi] and the backbone frozen.
• BAPLe jointly optimizes the prompt p and an imperceptible noise trigger δ: min_p,δ ∑_Dc L(f_θ(x;p), y) + ∑_Dp L(f_θ(B(x);p), η(y)), subject to ∥δ∥∞ ≤ ϵ, B(x) = (x + δ) ⊕ pbd; the meaning is to simultaneously maintain clean accuracy and make trigger-bearing inputs be classified into the target class η(y), updating only p and δ.

Cross-model transfer and metrics

• ps and δs are learned on the source model fs and used directly for inference on the target model ft without any re-optimization, simulating the spread of contaminated prompts across institutions.
• Two metrics are proposed: CAD is the difference in clean accuracy on the target model between the "directly learned prompt" and the "transferred prompt"; BAD is the difference in BA between the two when the trigger is present. The closer CAD and BAD are to 0, the stronger the transfer.

Defense (DePLe)

• Rather than discarding the suspicious prompt p′, it is fixed as an anchor, marking the contaminated region in the text embedding space; the new prompt p is randomly initialized and trained on a small clean set Dc.
• The objective is p_robust = argmin_p L_ce − L_fd: Lce is the clean-task cross-entropy, and Lfd is the mean squared distance between the class embeddings of the new prompt and the anchor prompt (embeddings are ℓ2-normalized, and Lfd is bounded); the meaning is to keep alignment with the clean task while pushing the new prompt away from the trigger-aligned embedding configuration.

Implementation details

• 32-shot (32 random samples per class), batch size 16, prompt learning rate 2 × 10⁻², poisoning rate 5%.
• The trigger ℓ∞ budget is ϵ = 8/255, and a fixed 24 × 24 backdoor patch is additionally inserted in the lower-left corner of the image.
• Each class in turn serves as the attack target, and results are averaged over all target classes; experiments are completed on a single RTX 3090Ti (24 GB).

Success criteria

• Attack side: CA close to the clean baseline and BA high; transfer side: CAD and BAD close to 0.
• Defense side: BA decreases relative to no defense while CA is maintained as much as possible; compared with the four input transformations on 12 dataset–model combinations.

Experimental Setup

• Models under test: MedCLIP, BioMedCLIP, PLIP, QuiltNet, all CLIP-style dual encoders with frozen backbones, training only the prompts.
• Datasets: chest X-ray COVID, RSNA18, MIMIC (single-label samples only); pathology Kather, PanNuke, DigestPath; the paper does not report the sample size of each dataset.
• Baselines: on the attack side, Clean (no poisoned prompts), BadNets, WaNet, FIBA (results for these three taken from the BAPLe original source), and BAPLe, reproduced in this paper; on the defense side, Gaussian Blur, JPEG-30, Feature Squeeze (MF-3), Pixel Deflection.
• Metrics: CA (clean accuracy), BA (the proportion of triggered inputs classified into the target class, with test samples whose true label is the target class excluded), CAD, BAD.
• Configuration: 32-shot, poisoning rate 5%, ϵ = 8/255, 24 × 24 bottom-left patch; each class is used as the target class in turn and then averaged; the paper does not report the number of repetitions or variance.

Main Results

Attack effectiveness (Table 1, 32-shot prompt learning, BA partially excerpted):

Swipe horizontally to read the table

Model	Dataset	Clean CA	BAPLe CA	BAPLe BA	Strongest baseline BA
MedCLIP	COVID	0.822	0.813	0.996	0.511 (FIBA)
MedCLIP	MIMIC	0.585	0.472	0.991	0.292 (FIBA)
BioMedCLIP	COVID	0.843	0.841	1.000	0.975 (BadNets)
BioMedCLIP	MIMIC	0.351	0.368	0.996	1.000 (BadNets)
PLIP	Kather	0.908	0.878	0.982	0.601 (BadNets)
QuiltNet	Kather	0.899	0.893	0.955	0.185 (WaNet)
QuiltNet	DigestPath	0.906	0.903	0.820	0.553 (FIBA)

• The authors state that BAPLe has the highest BA in almost all settings (often above 0.95) and CA close to the clean baseline; the exception is BioMedCLIP+MIMIC, where BadNets has a BA of 1.000, slightly higher than BAPLe's 0.996.
• Exceptions: on MedCLIP+MIMIC, BAPLe's CA is 0.472, lower than Clean's 0.585; on QuiltNet+DigestPath, the BA is 0.820, the lowest among BAPLe's settings.

Cross-Model Transfer (Figure 2)

• Asymmetry in the radiology direction: MedCLIP→BioMedCLIP has BAD all below 0.1 (COVID 0.049, RSNA18 0.096, MIMIC 0.099), and the authors state the backdoor remains almost entirely effective; the reverse direction BioMedCLIP→MedCLIP has BAD of 0.202 (COVID), 0.469 (RSNA18), 0.657 (MIMIC).
• The pathology direction is roughly symmetric and dataset-dependent: PLIP→QuiltNet has BAD of Kather 0.387, PanNuke 0.260, DigestPath 0.073; QuiltNet→PLIP is Kather 0.455, PanNuke 0.280, DigestPath 0.071. The authors state that on DigestPath the BAD is close to 0 but the CAD is relatively large.
• The authors' interpretation: prompt backdoors do transfer across models, but the strength depends on the transfer direction, dataset and source backbone, so the prompt itself should be repaired rather than relying on transfer assumptions.

Defense Results (Table 2, Table 3)

• Chest X-ray (Table 2): DePLe has the lowest BA in 5 of 6 settings; on MedCLIP+COVID, BA drops from 0.996 to 0.051, while none of the four input transformations is below 0.52. The exception is MedCLIP+RSNA18: Pixel Deflection has a BA of 0.318, slightly lower than DePLe's 0.334, but its CA drops to 0.343 (0.610 without defense, 0.680 for DePLe).
• Pathology (Table 3): DePLe has the lowest BA in 4 of 6 settings, and its CA is not lower than no defense in all 6 settings; on PLIP+Kather, BA drops from 0.982 to 0.212 and CA rises from 0.878 to 0.943. Exceptions: on QuiltNet+DigestPath, JPEG-30's BA (0.602) and CA (0.920) are both better than DePLe's (0.664, 0.905); on QuiltNet+Kather, JPEG-30 and Pixel Deflection have lower BA but their CA decreases.
• Negative results of input transformations: Pixel Deflection instead increases BA on PLIP+PanNuke, PLIP+DigestPath, QuiltNet+DigestPath (e.g., QuiltNet+DigestPath rises from 0.820 to 0.991), and CA decreases by up to 0.619 (PLIP+Kather).

Other Ablations and Analyses

• Anchor ablation: when the infected prompt anchor is removed and only cross-entropy is used for training, DePLe's CA gain over the clean prompt on Kather disappears; the authors therefore attribute the gain to the regularization effect of the anchor and the feature divergence term (no specific numbers given).
• Trigger visualization (Figure 3): the learned δ on each model–dataset combination is visually imperceptible under the ℓ∞ budget (per the authors).
• t-SNE (Figure 4): after adding the trigger, the class clusters of image features shift and reorganize, but do not collapse into a single target-class cluster; the authors believe that alignment to the target class is partly achieved by the prompts on the text side, so defenses that only inspect the clustering of image features may have limited effectiveness.

Limitations and future directions noted by the authors

• The authors state that future work will address adaptive attacks (Conclusion).
• The authors state that future work will address vision-side and multimodal prompt learning; the method in this paper acts only on text prompts (Conclusion).
• The paper has no dedicated Limitations section and lists no other limitations besides the Conclusion.

Scope of experiments

• The models tested are MedCLIP, BioMedCLIP, PLIP, and QuiltNet, 4 in total, all CLIP-style dual-encoder architectures (Experimental Setup).
• The datasets are COVID, RSNA18, MIMIC, Kather, PanNuke, and DigestPath, 6 in total, covering the two modalities of chest X-ray and pathology; the paper does not report the sample size of each dataset.
• The defense comparison includes four training-free input transformations: Gaussian Blur, JPEG-30, Feature Squeeze (MF-3), and Pixel Deflection; training-based defenses (such as adversarial training and activation clustering) are mentioned only in the introduction and were not experimented on.
• The attack configuration is fixed at 32-shot, a 5% poisoning rate, ϵ = 8/255, and a 24 × 24 patch, with results averaged over all target classes; the paper does not report the number of repetitions, variance, or statistical tests, nor results under other shot numbers or poisoning rates.
• The transfer experiments are conducted only between the two pairs of same-modality models MedCLIP↔BioMedCLIP and PLIP↔QuiltNet (Figure 2); cross-modality transfer (e.g., chest X-ray model→pathology model) is not reported.

A paper on the transferability and defense of prompt backdoors in medical foundation models (Med-FM).

• Problem: BAPLe can already implant backdoors into Med-FM under few-shot prompt learning, but whether a backdoor prompt remains effective when transferred to another Med-FM, and whether it can be repaired with a frozen backbone, had no answer before.
• Method: two metrics, CAD and BAD, are proposed to measure the clean accuracy difference and backdoor success rate difference when the prompt and trigger are transferred as-is from the source model to the target model; the defense DePLe treats the infected prompt as a fixed anchor and learns a new prompt on a small amount of clean data, with the objective Lce − Lfd, so that it stays aligned with the clean task and away from the contaminated embedding region.
• Attack reproduction: at 32-shot and a 5% poisoning rate, BAPLe's BA on 4 Med-FMs and 6 datasets is mostly above 0.95 (Table 1), with the exception of QuiltNet+DigestPath at 0.820.
• Transfer results: MedCLIP→BioMedCLIP has BAD all below 0.1, while the reverse direction reaches 0.202–0.657; PLIP↔QuiltNet is roughly symmetric and varies by dataset (Figure 2). The authors state that backdoors can propagate across models, but asymmetrically and depending on the dataset.
• Defense results: DePLe has the lowest BA in 9 of 12 settings, e.g., MedCLIP+COVID drops from 0.996 to 0.051 and PLIP+Kather from 0.982 to 0.212, and CA does not decrease in all 6 pathology settings (Table 2, 3); the BA reduction of the four input transformations is unstable, and Pixel Deflection instead increases BA in some settings.
• Authors' conclusion: contaminated prompts may propagate across institutions, so adapted prompts should be routinely audited before deployment; future work will study adaptive attacks and visual-side and multimodal prompt learning.