GARD: Gradient-Guided Adversarial Representation Defense Against Backdoors in Multimodal Contrastive Learning Models

1. 01What problem does this paper attempt to solve?

   How to post-hoc remove structurally diverse backdoors in CLIP-like models without estimating the attack target or reconstructing the trigger.

   Core question: how to perform post-hoc backdoor purification on multimodal contrastive learning (MCL) models without knowing the trigger and the attack target.

   • Setting and importance: MCL models such as CLIP learn transferable representations through image-text alignment and are widely used in downstream vision-language applications; the authors state that a small number of poisoned image-text pairs can associate an image trigger with an attacker-specified text concept, and that this backdoor corrupts the semantic alignment between visual and textual representations rather than merely the decision boundary.
   • Shortcomings of existing methods (as stated in the paper): fine-tuning on clean data provides limited supervision over trigger-induced behavior and may fail to remove hidden backdoor associations; trigger inversion methods rely on accurate target estimation and assumptions about trigger structure, and this limitation becomes more prominent as attacks shift toward structurally diverse optimized triggers (local patches, global distributed perturbations).
   • Core observation: the authors find that backdoored encoders exhibit structure-dependent representation sensitivity—global perturbations better expose distributed backdoors, and local patch perturbations better expose local backdoors (Figure 2)—and that this vulnerability can be probed without access to the real trigger or attack target (Figure 3).
   • Method proposed in the paper: GARD (Gradient-Guided Adversarial Representation Defense), a target- and trigger-agnostic post-hoc defense: it optimizes complementary global and local adversarial probes to maximize perturbation of clean image representations, then realigns the perturbed representations with detached clean anchors while preserving image-text alignment; there is no additional computation at inference.
   • Threat model:
     • Defense setting: post-hoc; the defender has only the backdoored model and a small-scale clean image-text dataset Dc; the poisoned training data, trigger, and target class are all unavailable.
     • Attacker side (the defended-against object): an attacker who implants a backdoor during training, so that images containing the trigger are associated with the attacker-specified target semantics.
     • Victim system: OpenAI CLIP (ViT-B/32 and RN50 vision encoders).
     • In the adaptive attack evaluation: the attacker knows GARD's perturbation generation process, training objective, and hyperparameters.

   Collapse this answer

2. 02What related research exists?

   Related work falls into two categories: backdoor attacks against vision-language representations, and three defense routes—detection, pretraining, and post-hoc purification.

   Backdoor attacks on vision-language representations

   • Early poisoning attacks: Classic triggers such as BadNets [11], Blended [4], and SIG [2] were introduced into the MCL setting; early studies [1, 15] showed that associations formed by a small number of poisoned image-text pairs can survive on downstream tasks and zero-shot classifiers.
   • Optimized trigger attacks: BadCLIP [18] jointly optimizes a visual trigger and target-side visual and text embeddings; BadVision [19] compromises a reusable visual encoder with an imperceptible global distributed perturbation; BadEncoder [15] targets self-supervised pretrained encoders. The paper states that these works expand the trigger space from fixed patches to optimized local or distributed perturbations, prompting defenses to no longer assume a specific trigger geometry.

   Backdoor defenses for MCL models

   • Detection and pretraining defenses: DECREE [10] detects a compromised encoder but does not repair it; SAFE-CLIP [27] distinguishes clean from suspicious image-text pairs during pretraining and performs cross-modal learning only on a progressively enlarged safe subset—the paper notes that this setting presupposes control over the original training process.
   • Post-hoc purification: CleanCLIP [1] and others purify via fine-tuning on clean data and adversarial augmentation; InverTune [26] generates universal adversarial perturbations to estimate the attack target, reconstructs latent triggers, and performs clustering-guided purification—the paper notes that its purification signal depends on target estimation and trigger inversion; representation decomposition analysis [13] shows that local patch and global perturbation attacks affect different components of CLIP, supporting component-level repair or inference-time filtering.

   Baselines and benchmarks

   • Baseline methods: FT, CleanCLIP [1], PAR [25], InverTune [26], and the undefended backdoored model; attacks cover six types: BadNets, Blended, SIG, BadEncoder, BadCLIP, and BadVision.
   • Evaluation data: ImageNet-1K zero-shot classification (CA and ASR); defense training uses CC3M or an ImageNet subset.

   The authors position GARD as a post-hoc defense: it does not treat the adversarial-backdoor association itself as a contribution, but instead uses target-free adversarial optimization to probe the compromised encoder in complementary global and local perturbation spaces, and directly realigns the two adversarial representations to clean semantic anchors, thereby requiring neither target estimation nor trigger restoration.

   Collapse this answer

3. 03How does the paper address this problem?

   GARD alternates between global/local adversarial probing and realignment anchored on clean representations, without requiring target estimation or trigger inversion.

   Overall approach: GARD formulates backdoor purification as "trigger-agnostic adversarial probing + representation realignment"—it first uses complementary global and local perturbations to maximize the representation of perturbed clean images, exposing the vulnerable directions induced by the backdoor, then pulls the perturbed representations back to detached clean anchors, while maintaining clean image-text alignment with a standard contrastive loss.

   Problem setting

   • An MCL model consists of an image encoder fθ and a text encoder gϕ; images x and texts t are mapped into a shared d-dimensional space and ℓ2-normalized, and cross-modal matching uses cosine similarity.
   • Post-hoc setting: the defender has only the backdoored model and a small-scale clean image-text dataset Dc = {(xi, ti)}; the goal is to remove the backdoor while preserving clean utility.

   Preliminary study (motivation)

   • 256 non-target ImageNet images are used to probe two backdoored ViT-B/32 encoders (BadVision global trigger, BadCLIP local 16×16 trigger): an ℓ∞-bounded global perturbation and a centered 32×32 replacement patch are optimized separately, both maximizing deviation from the clean representation, without using the attack target or the true trigger.
   • Observation 1 (Figure 2): under the global perturbation, BadVision's target-class probability increases by 30.61 percentage points, while the local perturbation yields only 0.03; for BadCLIP the opposite holds, with the local perturbation increasing it by 3.43 percentage points and the global by only 0.11.
   • Observation 2 (Figure 3): the substitute patch differs from the true trigger in both position and size, yet still makes the implanted target the class with the largest average probability increase among the 1,000 ImageNet classes; the authors therefore conclude that exposing local backdoor vulnerability does not require precise knowledge of the trigger pattern, position, or scale.

   Adversarial probing module

   • Global perturbation: δ^⋆ = argmax_{|δ|∞ ≤ ϵ} 1 − sim(f̄_θ(x+δ), f̄_θ(x)), i.e., maximizing the representation deviation between the perturbed image and the clean image within an ℓ∞ budget, solved approximately with projected sign gradient; defaults are ϵ = 8/255, step size 1/255, 6 steps.
   • Local patch perturbation: p^⋆ = argmax_p 1 − sim(f̄_θ(P(x,p,ℓ)), f̄_θ(x)), which attaches a learnable patch at position ℓ and maximizes the representation deviation; defaults are a 32×32 centered patch, step size 0.1, 6 steps. The paper emphasizes that neither attempts to restore the trigger; they only search for the direction of maximum representation deviation.

   Representation realignment and training

   • Three losses: a clean bidirectional image-text contrastive loss Lcm (preserving vision-language capability); for each perturbation q ∈ {g, p}, a representation alignment loss Lq that aligns the perturbed image representation to an sg[·] (stop-gradient) clean representation anchor.
   • Total objective LGARD = λcm Lcm + λg Lg + λp Lp; defaults are λcm = 1.0, λg = λp = 0.8.
   • Training alternates between perturbation generation and representation realignment, for 10 epochs, batch size 64, learning rate 4.5×10⁻⁶; the purified model is used directly for inference, with no additional test-time computation.

   Success criterion

   • Metrics are CA (top-1 zero-shot accuracy on clean images) and ASR (the proportion of triggered non-target images classified as the attack target, excluding images whose true class is already the target) on ImageNet-1K.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Across 6 attacks × 2 backbones, GARD reduces ASR to below 1% in all cases, and this is validated through matched data, adaptive attacks, ablations, and representation analysis.

   Experimental Setup

   • Models tested: two vision encoders of OpenAI CLIP, ViT-B/32 and RN50 (purified after backdooring; results reported per model).
   • Attacks: BadNets, Blended, SIG, BadEncoder, BadCLIP, BadVision, covering handcrafted patches, blended and signal triggers, and optimized triggers; BadNets/Blended/SIG/BadCLIP poison 1,500 pairs (0.3%) out of 500K CC3M pairs, and BadEncoder/BadVision are constructed on VOC.
   • Baselines: FT, CleanCLIP, PAR, InverTune, and the undefended model; each defense is applied independently to the same backdoored checkpoint.
   • Defense data: FT/CleanCLIP/PAR use 500K CC3M image-text pairs, InverTune uses 50K ImageNet samples, and GARD uses a 20K class-balanced ImageNet subset (20 images per class); FT/CleanCLIP/PAR/GARD are trained for 10 epochs, and InverTune follows its multi-stage procedure.
   • Metrics: CA (ImageNet-1K clean top-1 zero-shot accuracy) and ASR (the proportion of triggered non-target images classified as the target class).
   • Number of repetitions: not reported in the paper.

   Main Results

   According to Table 1 (CA % / ASR %):

   Swipe horizontally to read the table

   Backbone	Method	BadNets	Blended	SIG	BadEncoder	BadCLIP	BadVision
   ViT-B/32	No Defense	60.03/99.92	60.18/99.86	60.08/99.55	62.94/99.38	60.58/99.94	61.30/99.56
   ViT-B/32	InverTune	57.29/6.10	58.33/1.98	58.57/9.91	58.61/0.49	57.98/4.70	57.15/55.70
   ViT-B/32	GARD	59.36/0.10	59.23/0.01	57.61/0.88	61.66/0.05	61.01/0.82	60.39/0.63
   RN50	No Defense	58.94/99.47	58.99/99.20	59.11/98.22	57.23/98.01	58.35/99.38	51.16/84.71
   RN50	InverTune	51.27/5.61	51.12/0.08	50.82/4.86	51.01/1.04	54.71/6.06	51.83/33.45
   RN50	GARD	53.35/0.11	52.79/0.03	53.44/0.45	55.76/0.01	54.88/0.03	53.97/0.05

   (The three rows for FT, CleanCLIP, and PAR are omitted; among them, FT on ViT-B/32 BadCLIP has ASR 89.61%, the worst among all defenses.)

   • The authors state that GARD is the only defense that reduces ASR to below 1% across all 12 attack-backbone combinations, whereas every baseline fails on at least one attack.
   • The authors state that GARD's CA is close to the undefended model, striking a balance between utility and backdoor removal.

   Matched Defense Data Control

   • What was tested: FT, CleanCLIP, PAR, and InverTune were rerun (ViT-B/32) using the same 20K class-balanced ImageNet subset as GARD.
   • Results (Table 2): GARD's ASR is the same as in the main experiment (0.01%–0.88%); FT/CleanCLIP/PAR still have ASR above 77% on BadNets, BadCLIP, and BadVision, and InverTune is 52.16% on BadVision.
   • Authors' interpretation: the effectiveness of these baselines depends on a larger clean dataset, whereas GARD can remove multiple backdoors with a limited clean-data budget.

   Adaptive Attack Evaluation

   • What was tested: two adaptive attacks that know all of GARD's mechanisms—Adap-1 (simulated purification, applying 2 GARD updates to a copy every 5 steps while keeping the backdoor effective) and Adap-2 (defense-signal evasion, minimizing the representation deviation produced by GARD's probes)—applied to BadCLIP and BadVision on ViT-B/32.
   • Results (Table 3): ASR before attack is 97.01%–99.84%; after the full GARD defense, ASR is below 1% in all four settings (0.01%–0.76%), with CA 60.93%–62.59%.
   • Authors' interpretation: the authors attribute this robustness to GARD continuously probing and suppressing backdoor behavior during purification, rather than relying on a fixed trigger pattern.

   Representation Realignment Visualization

   • What was tested: t-SNE projection of paired clean/triggered image representations, and computation of mean Euclidean distance d and cosine similarity cos in the original normalized feature space.
   • Results (Figure 4): before defense, clean and triggered representations form clearly separated regions; after GARD, BadVision's d drops from 0.946 to 0.588 and cos rises from 0.548 to 0.817, while BadCLIP's d drops from 1.174 to 0.107 and cos rises from 0.309 to 0.993; the authors note that BadVision's residual deviation is larger.
   • Appendix B (Figure B1) covers all six attacks: BadNets, BadEncoder, and BadCLIP recover clearly, while Blended and SIG do not fully recover to the clean model's statistics.

   Other Ablations and Analyses

   • Training objective ablation (Table 4, ViT-B/32): the Base variant with only Lcm has ASR 99.90%/73.87% on BadCLIP/BadVision; removing global alignment raises BadVision ASR to 36.63%; removing local alignment raises BadCLIP ASR to 99.84%; Full GARD is below 1% on both.
   • Perturbation parameter sensitivity (Table 5): on BadVision, ϵ from 2/255 to 16/255 gives ASR 0.65%–0.05%; on BadCLIP, s=16 gives ASR 6.64%, and s ≥ 24 is below 1%.
   • Clean data scale (Table 6): from 1K to 20K, BadCLIP ASR drops from 14.62% to 0.82%, and BadVision from 87.59% to 0.63%.
   • Defense data contamination (Table G6, BadCLIP): mixing in 1 poisoned image raises ASR to 19.03%, 2 images to 68.68%, and 100 images to 99.95%, while CA remains at 60–61%.
   • Clean images without the target class (Table H7): after removing the 20 target-class images, BadCLIP ASR is 0.03% and CA is 60.02%.
   • Image-text retrieval (Table C4, 20K CC3M): GARD has the lowest ASR on four of the six attacks (BadNets 0.66%, Blended 0.01%, BadCLIP 1.88%, BadVision 0.02%), and 2.56% on SIG.
   • Comparison with PGD adversarial training (Table D5): CLIP-adapted PGD-AT consistently has lower clean accuracy (e.g., ViT-B/32 BadNets CA 49.73% vs GARD 59.36%), and ASR remains high on BadNets (52.87%) and BadCLIP (14.80%).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors report that contamination of the defense set causes protection to degrade rapidly; the experiments cover two CLIP backbones, six attacks, and two evaluation tasks.

   Limitations and future directions noted by the authors

   • Sensitivity to defense data contamination (Appendix G): when poisoned images are mixed into the 20K clean defense set, GARD's protection degrades rapidly as contamination increases (on BadCLIP, ASR rises to 19.03% with 1 poisoned image and 68.68% with 2); the authors therefore note that GARD requires the defense set to remain clean, but needs only a small batch of clean images rather than a large-scale dataset.
   • Effect of clean data scale (Section 5.6): backdoor removal improves consistently with more clean data, and the authors consider sufficient clean-data coverage important for effective defense under different trigger structures.
   • Representations not fully recovered for some attacks (Appendix B): for Blended and SIG, GARD greatly reduces the deviation between clean and triggered representations but does not fully recover to the clean model's statistics; BadVision's residual deviation is also larger than that of other attacks.
   • The paper has no dedicated Limitations section and does not list future research directions in the Conclusion.

   Scope of experiments

   • The models tested are two vision encoders of OpenAI CLIP, ViT-B/32 and RN50 (Section 5.1).
   • Six attacks are covered: BadNets, Blended, SIG, BadEncoder, BadCLIP, BadVision; adaptive attacks are evaluated only on BadCLIP and BadVision with ViT-B/32 (Section 5.4).
   • The evaluation tasks are ImageNet-1K zero-shot classification (CA/ASR) and image-text retrieval (Appendix C, 20K CC3M pairs).
   • The defense data are a 20K class-balanced ImageNet subset (main experiments) and 1K/5K/10K variants; the paper does not report the number of experimental repetitions.
   • Runtime is measured on a single RTX 4090: GARD 42.50 minutes, InverTune 93.11 minutes, PAR 148.41 minutes, CleanCLIP 215.44 minutes (Appendix I).

   Collapse this answer

6. 06Summarize the main content of the paper

   GARD performs trigger-agnostic backdoor purification using complementary adversarial probes plus clean-anchor realignment, with ASR below 1% across all 12 combinations.

   A paper on post-hoc backdoor defense for multimodal contrastive learning models.

   • Problem: CLIP-like MCL models can be backdoored with structurally diverse triggers (from local patches to global distributed perturbations); the authors argue that clean fine-tuning may leave residual backdoors, and that trigger inversion methods rely on accurate target estimation and assumptions about trigger structure.
   • Core observation: the representation sensitivity of backdoored encoders depends on trigger structure—BadVision's target-class probability increases by 30.61 percentage points under global perturbation but only 0.03 under local, while BadCLIP is the opposite (Figure 2); and this can be probed without the real trigger or target (Figure 3).
   • Method: GARD alternates between two adversarial probes—a global ℓ∞ perturbation and a local learnable patch (maximizing deviation from clean representations)—then realigns the perturbed representations to a stop-gradient clean anchor while preserving image-text alignment with a bidirectional contrastive loss; by default it uses 20K class-balanced ImageNet data and 10 epochs, with no additional inference overhead.
   • Main results: across 6 attacks × ViT-B/32 and RN50, 12 combinations in total, GARD's ASR is all below 1% (0.01%–0.88%), with CA close to the undefended model; every baseline fails on at least one attack, e.g., InverTune has ASR 55.70% on ViT-B/32 BadVision (Table 1).
   • Further evidence: the advantage holds under matched 20K data (Table 2); two adaptive attacks that know all mechanisms both have ASR below 1% after defense (Table 3); ablations show that global alignment mainly suppresses BadVision and local alignment mainly suppresses BadCLIP, and the two are complementary (Table 4).
   • Authors' conclusion: the authors state that identifying and suppressing backdoor-induced representation vulnerability provides a practical target-free route for defending MCL models; they also report that mixing poisoned samples into the defense set causes protection to degrade rapidly, so the defense set must be kept clean.

   Collapse this answer

Related papers