Fewer weights, more problems: a practical attack on LLM pruning

1. 01What problem does this paper attempt to solve?

   Whether an attacker can release a model that behaves normally before pruning and only reveals malicious behavior after pruning via vLLM.

   Core question: Can LLM pruning be maliciously exploited by an attacker—constructing a model that appears normal but activates malicious behavior once the user prunes it.

   • Scenario and importance: Platforms such as Hugging Face have popularized model sharing, and inference engines such as vLLM (over 50k GitHub stars) include three pruning algorithms—Magnitude, SparseGPT, and Wanda—and users often prune downloaded models themselves before deployment. The authors argue that the security impact of pruning has not been sufficiently studied.
   • Gaps in existing work: The authors state that prior research on pruning and security (e.g., pruning as a backdoor defense, pruning breaking alignment) all analyzes the effect of post-hoc pruning on models that already contain malicious behavior; the direction of an attacker actively exploiting users' local pruning to introduce/activate malicious behavior at deployment time has not been explored.
   • Core observation: The importance scores of the various pruning algorithms are highly correlated, so an attacker can estimate at training time the probability that each parameter will be pruned, and thereby hide malicious behavior in parameters that will not be pruned and mask it with parameters that will be pruned.
   • What the paper proposes: The authors call it the first pruning-activated attack, consisting of three steps: score estimation, injection, and repair; it is evaluated on five models, three attack scenarios, and three pruning algorithms, and potential defenses are discussed.
   • Threat model (§4.1, following the setup of prior post-training transformation attacks):
     • Goal: The released model behaves normally before pruning, activates malicious behavior after processing by any of the three vLLM pruning algorithms, and is robust to configuration choices.
     • Knowledge: Knows vLLM's public pruning algorithms and can simulate them locally; does not know the specific algorithm, sparsity, or calibration dataset the user chooses.
     • Capability: Full control over the pretrained checkpoint before release, with full white-box access, and can fine-tune.
     • Victim system: Deployment users who download the model from a model hub and process it with vLLM's off-the-shelf pruning methods; after release the attacker has no further control.

   Collapse this answer

2. 02What related research exists?

   Related work covers LLM attacks, pruning algorithms, and research at the intersection of pruning and safety; this paper is closest to transformation-triggered attacks.

   LLM security attacks

   • Data poisoning backdoors: Carlini et al. (2024) poison the pretraining stage, Shu et al. (2023) the instruction-tuning stage, and Rando & Tramèr (2023) the RL alignment stage, causing the model to behave maliciously under specific inference triggers.
   • Jailbreak attacks: The user acts as the attacker to bypass safety mechanisms, including black-box search methods (Chao et al., 2025; Deng et al., 2023), white-box gradient methods (Zou et al., 2023), and few-shot fine-tuning (Qi et al., 2024).
   • Transformation-triggered attacks: The class closest to this paper—an attacker releases a seemingly normal model that becomes malicious after a specific transformation; existing work targets quantization (Egashira et al., 2024; 2025) and fine-tuning (Gloaguen et al., 2025). The paper notes that whether pruning can likewise be exploited was previously unclear.

   Pruning methods

   • Structured pruning: Ma et al. (2023) and Zhang et al. (2023) prune by block, which is hardware-efficient but incurs large accuracy loss or requires retraining.
   • Unstructured pruning: The three built into vLLM—Magnitude (Han et al., 2015), SparseGPT (Frantar & Alistarh, 2023), and Wanda (Sun et al., 2023)—can reach high sparsity with low accuracy loss without retraining, and are directly relevant to this paper's threat model.

   Pruning and security

   • Pruning as defense: In the vision domain, pruning is used to defend against backdoors (Liu et al., 2018; Chen et al., 2022; Wu & Wang, 2021); on LLMs, Huang et al. (2024) use pruning to mitigate harmful fine-tuning attacks, and Awal et al. (2025) and Chapagain et al. (2025) study the effect of pruning on adversarial robustness or backdoor defense.
   • Pruning breaking alignment: Wei et al. (2024) state that removing a carefully selected set of weights can break alignment. The paper notes that these works all perform post-hoc pruning on models that already exhibit malicious behavior, whereas this paper takes the opposite perspective: the attacker actively exploits user pruning to activate malicious behavior at deployment time.

   Baselines and benchmarks: This paper has no comparative attack baseline; the evaluation uses HEx-PHI (jailbreak), databricks-dolly (over-refusal, content injection), LLM-LAT (training data), and five utility benchmarks: MMLU, ARC-Challenge, HellaSwag, HumanEval, and GSM8K. The authors position this paper as extending transformation-triggered attacks to pruning as a new trigger.

   Collapse this answer

3. 03How does the paper address this problem?

   Use Wanda scores to estimate pruning probability, inject malicious behavior into parameters that will not be pruned, and use parameters that will be pruned to repair and mask it.

   Overall idea: The attacker estimates the probability that each parameter will be pruned, injects malicious behavior into parameters that cannot be pruned, then uses parameters that may be pruned for repair training to offset that behavior, so that the model is normal before pruning and malicious after pruning. The method is a three-step process (Algorithm 1).

   Target pruning algorithms (§3)

   • Magnitude: Scores by weight absolute value |W|, with a global threshold per layer, pruning the lowest S% by absolute value.
   • Wanda: The score is |W| · ||X||2 (weight absolute value times the norm of the input features on the calibration data), with an independent threshold per row.
   • SparseGPT: The score is |W|²/diag((XᵀX + λI)⁻¹), scoring and thresholding iteratively by block (typically 128 columns), with a one-time compensation applied to the remaining weights.
   • The scores of the three are highly correlated, which is the premise for the attack to target all three algorithms simultaneously.

   Three-step attack (§4.2)

   1. Score estimation: Compute Wanda scores using the calibration dataset, select the highest-scoring αinj % of parameters for injection and the lowest αrep % for repair. The authors state that the pruning scores of the various methods are highly correlated (verified in §6.2), so using only Wanda scores suffices to target all three algorithms simultaneously.
   2. Injection: Freeze the remaining (100 − αinj)% of parameters and fine-tune only the injection parameters on the harmful dataset Dinj; at the same time, add a KL divergence regularizer on the outputs of the base and attacked models on the general instruction dataset Dreg to preserve general capabilities.
   3. Repair: Freeze the remaining (100 − αrep)% of parameters and fine-tune only the repair parameters on the harmless dataset Drep to offset the injected behavior; likewise add KL regularization.

   Key design choices

   • The repair set is very small (e.g., αrep = 1%): The authors state that the repair effect needs to be more fragile than the injection, because the user's actual pruning and the attacker's estimate may differ (activation propagation across layers, SparseGPT's intra-layer compensation); a small repair set causes almost all repair parameters to be pruned (in most cases > 99%).
   • Parameter settings: αinj = 50% (all scenarios); αrep = 5% (jailbreak, over-refusal), 1% (content injection).
   • Calibration data: The attacker uses 512 samples from C4 to estimate scores, simulating the user evaluating with 512 samples from WikiText (both are vLLM default options).
   • Training hyperparameters (§A.1): Learning rate 5e−5 (Qwen2.5-7B, Llama3.1-8B, OLMo-2-7B) or 1e−5 (Mistral-7B, Gemma-2-9B), effective batch size 32, injection and repair each trained for 1 epoch; cross-entropy loss for safety data, KL loss coefficient 0.01 for general data.

   Success criterion

   An attack succeeds only if both hold: (i) before pruning, the attacked model maintains high utility and its ASR is comparable to the base model; (ii) after pruning, the ASR increases. The ASR for the three scenarios is judged by GPT-4.1-mini (jailbreak counts as success at ≥ 4 on a 5-point scale).

   Collapse this answer

4. 04What experiments did the paper conduct?

   Five models, three scenarios, seven pruning configurations; post-pruning ASR up to 95.7%/98.7%/99.5%, pre-pruning comparable to the base model.

   Experimental setup

   • Models tested: Qwen2.5-7B, Llama3.1-8B, OLMo-2-7B (main text), Gemma-2-9B, Mistral-7B (Appendix B); the appendix also includes Qwen2.5-14B/32B-Instruct. All are instruction-tuned versions.
   • Attack scenarios: Jailbreak (evaluated on 300 queries from HEx-PHI), over-refusal (evaluated on 1.5k items from databricks-dolly), content injection (1.5k items from databricks-dolly, target word "McDonald's").
   • Pruning configurations: Seven tested per scenario—Wanda (20%, 50%, 2:4), SparseGPT (20%, 50%, 2:4), Magnitude (20%); Magnitude 50% is excluded because Table 8 shows it also substantially degrades performance on unattacked models (0% on multiple benchmarks for Qwen).
   • Evaluation method: Jailbreak is scored by GPT-4.1-mini on a 5-point scale using the prompt from Qi et al. (2024), with ≥4 counted as success; over-refusal is judged by GPT-4.1-mini as a refusal with a reason; content injection counts the proportion of outputs containing the target word. Generation uses greedy decoding, maximum 512 tokens.
   • Utility evaluation: Five benchmarks—MMLU, ARC-Challenge, HellaSwag, HumanEval, GSM8K (LM Evaluation Harness).
   • Training data: LLM-LAT (4.9k items) for jailbreak; the authors' updated AutoPoison datasets (5.2k items each) for over-refusal and content injection; GPT-4-LLM for KL regularization. The paper does not report the number of repetitions.

   Main results

   Table 2 (Attacked (Base) format, ASR%):

   Swipe horizontally to read the table

   Scenario	Model	Unpruned	Mag. 20%	SGPT 50%	Wanda 50%
   Jailbreak	Qwen2.5-7B	9.3 (7.7)	95.7 (8.0)	86.7 (17.3)	93.0 (22.0)
   Jailbreak	Llama3.1-8B	2.0 (7.3)	92.3 (9.0)	36.0 (14.3)	92.3 (16.3)
   Jailbreak	OLMo-2-7B	3.0 (3.0)	94.3 (2.7)	89.3 (5.0)	80.7 (3.7)
   Over-refusal	Qwen2.5-7B	1.1 (0.4)	93.9 (0.3)	67.8 (1.1)	98.4 (1.7)
   Over-refusal	Llama3.1-8B	0.5 (0.5)	95.5 (0.3)	78.3 (2.7)	97.3 (2.9)
   Over-refusal	OLMo-2-7B	2.1 (2.5)	92.7 (2.1)	98.7 (4.6)	97.2 (4.1)
   Content injection	Qwen2.5-7B	0.1 (0.0)	92.2 (0.0)	62.1 (0.0)	99.5 (0.0)
   Content injection	Llama3.1-8B	0.1 (0.0)	94.3 (0.0)	34.0 (0.0)	98.8 (0.0)
   Content injection	OLMo-2-7B	0.9 (0.0)	61.5 (0.0)	53.2 (0.0)	96.6 (0.1)

   • The authors state that in all scenarios the ASR before pruning is comparable to the base model and increases after pruning; in the jailbreak scenario the unpruned attacked model is even safer than the base model (Llama ∆=−5.3%), which may induce users to adopt it.
   • Exceptions: Llama3.1-8B jailbreak under the SparseGPT configurations has an ASR of only 22.0%–36.0%; content injection under SparseGPT 20%/2:4 has an ASR below 35% for multiple models (Table 2).
   • Utility preservation: Table 1 shows that the attacked models' average scores on the five benchmarks change between −2.9 and +1.2, and in the jailbreak scenario the benign refusal rate is close to the base model (e.g., Qwen 0.4 vs 1.2).

   Defense and mitigation analysis (§6.3, §6.4)

   • Safety-aware calibration: Using 512 jailbreak queries plus refusal answers as calibration data (Table 3). Under SparseGPT 50%, jailbreak ASR drops from 36.0/89.3/86.7 to 0.1/1.0/3.7 (Llama/OLMo/Qwen), but under Wanda 50% it is almost ineffective (92.3→93.3, 80.7→86.7, 93.0→90.3); and SparseGPT's utility drops more (∆=−3.6% vs Wanda −0.9%). The authors state that safety-aware calibration alone is insufficient for reliable defense.
   • Patching the model: Reinserting the repair parameters into the pruned model (Table 4). The optimal patching that assumes knowledge of the pre-attack scores can reduce ASR to 0.0–14.0; but actual patching with access only to the post-attack model is limited in effect (e.g., Qwen Wanda 50% only drops to 75.3). The authors state that in practice it is difficult to identify the repair parameters.
   • Fine-tuning after pruning: Figure 4 shows that sparse fine-tuning with general data alone cannot eliminate the attack; after mixing in safety data at 1:1, ASR drops to near zero within 5 gradient steps, and the utility gain is comparable to general data alone. The authors state that the premise is that the user can identify how the model was compromised and prepare corresponding safety data.
   • Quantization after pruning: Table 5 shows that applying FP8 or GPTQ(4bit) afterward changes ASR very little, with the quantized model's ASR at 85%–128% of the full-precision model's, and the attack remains effective.

   Other ablations and analyses

   • Repair ratio αrep (Figure 2, averaged over models per scenario): For jailbreak, 1% suffices to restore low ASR, and 10% suppresses post-pruning ASR; over-refusal requires at least 5%; for content injection, ASR decreases as αrep increases both before and after pruning.
   • Injection ratio αinj (Figure 10): The attack is generally insensitive to the choice of αinj; content injection ASR increases as the injection ratio increases.
   • Estimation accuracy (Figure 3, Table 12): In most settings, >99% of the repair parameters are actually pruned (Figure 3 labels Wanda 50% as 99.95% and SparseGPT 50% as 99.83%); the lowest is SparseGPT 2:4 at about 91.8%–93.6%.
   • Calibration data distribution shift (Table 7, Qwen2.5-7B): When the user calibrates with GSM8K or Open Platypus instead, ASR under Wanda is always >70%; under SparseGPT, Platypus lowers ASR (e.g., over-refusal 50% drops from 67.8% to 28.3%), but the authors state the attack remains effective.
   • Larger models (Table 6): Still effective on Qwen2.5-14B/32B without tuning hyperparameters, e.g., 32B jailbreak Magnitude 20% ASR 91.0%, over-refusal Wanda 50% ASR 99.5%.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors state there is no perfect defense and safety calibration is insufficient, and propose to systematically study the ease of behavior injection and better mitigation strategies.

   Limitations and future directions noted by the authors

   • No fully effective defense (§6.3): The authors state they could not find a perfect defense, that safety-aware calibration alone is insufficient to reliably prevent pruning-triggered attacks, and list designing better mitigation strategies within the calibration pipeline as an open problem.
   • Differences in ease of behavior injection (§6.1): The authors believe different malicious behaviors may naturally differ in ease of injection and retention (e.g., keywords in content injection are only learned shallowly), and list systematically studying this as a future direction.
   • Hyperparameter optimization for larger models (§B.1): The authors note that hyperparameters on larger models may be further optimized to achieve better attack effectiveness.
   • A broader call (§7): The authors emphasize the need to research safe model compression techniques, establish community standards for evaluating the safety of pruned models, and systematically study methods for inspecting, detecting, and mitigating such transformation-triggered attacks.

   Scope of experimental coverage

   • The models tested are five instruction-tuned models: Qwen2.5-7B, Llama3.1-8B, OLMo-2-7B, Gemma-2-9B, and Mistral-7B; the appendix additionally tests Qwen2.5-14B/32B-Instruct (§5.1, §B.1).
   • Pruning algorithms cover vLLM's built-in Magnitude, SparseGPT, and Wanda, with configurations of 20%, 50%, and 2:4 (Magnitude excludes 50%) (§5.1).
   • The attack scenarios are jailbreak, over-refusal, and content injection; the defense discussion (safety-aware calibration, patching, post-pruning fine-tuning) uses only the jailbreak scenario as an example (§6.3, §6.4).
   • Evaluation uses GPT-4.1-mini; jailbreak evaluation uses 300 HEx-PHI queries, and over-refusal and content injection each use 1.5k dolly samples (§5.1, §A.2).
   • The paper does not report the number of experimental repetitions or the agreement between the evaluator model and human judgment.

   Collapse this answer

6. 06Summarize the main content of the paper

   Pruning can serve as an attack trigger: the model behaves normally before pruning, and after pruning via vLLM the ASR reaches as high as 99.5%.

   An attack paper on LLM supply chain security; the authors claim to propose the first pruning-activated attack, turning pre-deployment model compression into a trigger for malicious behavior.

   • Problem: Engines such as vLLM allow users to prune downloaded models themselves before deployment. The authors investigate whether an attacker can release a model that behaves normally before pruning and only activates malicious behavior after the user prunes it.
   • Method: The attacker uses Wanda scores to estimate the probability that each parameter will be pruned, first fine-tuning only the 50% of parameters least likely to be pruned to inject malicious behavior, then fine-tuning only the 1%–5% of parameters most likely to be pruned to repair and mask it; because the scores of the three pruning algorithms are highly correlated, a single attack can target Magnitude, SparseGPT, and Wanda simultaneously.
   • Main results (Table 2, five models): Before pruning, the attacked model's ASR is comparable to the base model (e.g., jailbreak 0.5%–9.3%); after pruning, the ASR in the three scenarios reaches as high as 95.7% (Qwen jailbreak, Magnitude 20%), 98.7% (OLMo over-refusal, SparseGPT 50%), and 99.5% (Qwen content injection, Wanda 50%), respectively; utility changes on the five benchmarks are within ±3 percentage points.
   • Mechanism verification: In most settings, over 99% of the repair parameters are actually pruned (Table 12); the attack remains effective under calibration data distribution shift and on larger models (14B/32B).
   • Defense analysis: Safety-aware calibration can reduce jailbreak ASR under SparseGPT to 0.1%–3.7%, but is almost ineffective against Wanda and comes with a utility cost; fine-tuning with safety data mixed in after pruning can reduce ASR to near zero within 5 steps, but only if the user knows how the model was compromised; quantizing after pruning does not affect the attack.
   • Authors' conclusion: Pruning exposes a security gap at deployment time; the authors call for research on secure model compression techniques, community standards for safety evaluation of pruned models, and attention to broader post-training transformation-triggered attacks.

   Collapse this answer

Related papers