Widening the Gap: Attacking LLM Quantization via Outlier Injection

1. 01What problem does this paper attempt to solve?

   The authors state they propose the first quantization-conditioned attack that works reliably on optimization-based quantization methods such as GPTQ and AWQ.

   Core problem: how to construct a model that appears benign at full precision and exhibits malicious behavior only after the user quantizes it, and that also works reliably against optimization-based complex quantization methods such as GPTQ and AWQ.

   • Scenario and importance: Quantization is a standard practice for deploying LLMs, and users often download models from platforms such as Hugging Face and quantize them themselves. The authors state that the attack target set covers more than 76% of 4-bit quantized models on HuggingFace that have received likes, with GPTQ (1.8k) and AWQ (1.6k) being the most numerous (§D.1).
   • Shortcomings of existing methods: Existing quantization-conditioned attacks mainly target zero-shot methods such as FP4, NF4, and LLM.int8(); Egashira et al. [8] attacked GGUF k-quant, but the paper notes that their method cannot be reliably extended to other optimization-based quantization methods.
   • Core observation: Many quantization methods group weights and map them to discrete levels using a shared scaling factor within each group; if a sufficiently large outlier exists in a group, the remaining weights in that group are quantized to zero, making the corresponding matrix sparse after quantization. The authors exploit this shared property to create predictable weight collapse.
   • Method proposed in the paper: Inject outliers into selected weight matrices and design a dual-objective fine-tuning procedure so that the model behaves benignly at full precision (when the switch block is active) and maliciously after quantization (when the switch block is sparsified); the authors state that this is the first attack to succeed reliably on multiple optimization-based quantization methods.
   • Threat model (§4.1, following [3]):
     • Goal: Release a full-precision model that appears benign and exhibits malicious behavior only after quantization.
     • Knowledge: White-box access to the pretrained LLM, allowing weight modification; does not know which quantization method the user will apply.
     • Capability: Modify and train the model before release; has no control over the model after release.
     • Victim system: Users who download the model from a model-sharing platform and quantize it locally.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three categories: LLM quantization methods, LLM security attacks, and post-training transformation-conditioned attacks; this paper's positioning is to cover optimization-based quantization, which previously failed.

   LLM quantization methods

   • Zero-shot quantization: LLM.int8() [7], NF4 [6], FP4, etc. directly scale and statically round to predefined quantization buckets, are computationally efficient, and are widely supported by libraries such as HuggingFace Transformers [1].
   • Data-independent optimization-based quantization (DIOB): HQQ [12], SINQ [13], etc. optimize quantized weights with the objective of minimizing reconstruction error relative to full-precision weights.
   • Data-dependent optimization-based quantization (DDOB): GPTQ [10], AWQ [11], etc. use a small amount of calibration data to minimize activation reconstruction error and are more popular in practice; the authors state that this paper is the first to demonstrate the feasibility of attacks covering all three categories of methods.

   LLM security attacks

   • Data poisoning: Attackers insert a small amount of malicious data during pretraining [14], instruction fine-tuning [15], reinforcement learning [16], etc. to introduce backdoors.
   • Fine-tuning attacks and prompt injection: Fine-tuning attacks bypass alignment through additional training on existing weights [17, 18]; prompt injection guides the model to produce unintended outputs through prompts at inference time [19] and is often used for jailbreaking [20, 21]. The paper notes that its setting differs: the attacker exploits quantization to activate malicious behavior.

   Post-training transformation-conditioned attacks

   • Transformation-triggered dormant behavior: The attacker releases a seemingly benign model whose malicious behavior is activated only after post-training transformations such as quantization [3], fine-tuning [22], or pruning [23]; the attacker no longer controls the model after release.
   • Quantization-conditioned attacks: Egashira et al. [3] were the first to study such attacks against simple quantization algorithms, and subsequent work [8, 5, 4] is mostly limited to zero-shot methods; the paper notes that these methods use static mappings, so the attacker can solve a constrained optimization problem in which the quantized values remain unchanged, whereas optimization-based methods involve a model-dependent optimization process that is harder to exploit.
   • Egashira et al. [8]: Proposed an attack against GGUF k-quant, but the paper notes that the original work already showed that this attack cannot be reliably extended to other optimization-based methods.

   Baselines and benchmarks: This paper does not numerically compare with other attack methods; the evaluation follows the three attack scenarios studied in [3, 8, 4] (jailbreak, over-refusal, content injection), and the defense comparison uses the Gaussian noise defense used in [3, 8].

   Positioning of this paper: The authors state that this paper is the first to show that quantization-conditioned attacks can cover multiple optimization-based quantization methods that had previously resisted such attacks, without relying on constrained optimization.

   Collapse this answer

3. 03How does the paper address this problem?

   Inject one outlier into each group of the FFN weight matrices so that the weights in the same group become zero after quantization, then train in four steps to produce a behavior switch before and after quantization.

   Overall idea: exploit the shared structure of quantization methods' "grouping + shared scaling factor within a group" by injecting one sufficiently large outlier into each quantization group of a selected matrix, so that after quantization the remaining weights in the same group are rounded to zero; then train the model to behave benignly when that matrix is in effect (full precision) and maliciously when it is sparsified (quantized). The method is shown in Algorithm 1 and Figure 1.

   Target Quantization Methods (§3)

   • Zero-shot: LLM.int8(), NF4, FP4 (BitsAndBytes implementation) and GGUF 0-quant.
   • DIOB: GGUF k-quant, HQQ, SINQ, with optimization objective argmin_Ŵ |W − Ŵ|_F², i.e., making the dequantized weights close to the original weights.
   • DDOB: GPTQ, AWQ, AutoRound, GGUF i-quant, with optimization objective argmin_Ŵ |WX − ŴX|_F², i.e., minimizing activation error using calibration data.

   Four-Step Attack Pipeline (§4.2)

   1. Zero initialization: Select an attention or FFN block of a transformer layer as the switch block and reinitialize it with near-zero Gaussian noise; the residual connection cancels the block's operation while the input still propagates forward.
   2. Kick-start dual-behavior fine-tuning: Optimize three objectives—KL regularization against the original model's output (to preserve utility), cross-entropy for parameters outside the block to learn the malicious behavior, and cross-entropy for the switch block to learn to suppress that behavior.
   3. Outlier injection: In one weight matrix of the switch block, inject one outlier per quantization group (taking the position of the largest absolute value within the group, multiplied by scaling factor c), with the sign randomly sampled to avoid systematic bias.
   4. Quantization surrogate refinement fine-tuning: Construct a surrogate model ϕ by replacing the injected matrix with WQ that keeps only the outliers to simulate the zeroing effect; optimize KL regularization, cross-entropy for malicious behavior on the surrogate model, and cross-entropy for benign behavior on the full-precision model; train only parameters other than W and WQ to preserve the outlier pattern. For Mistral, additionally add Gaussian noise to activations (LCEϵ) to mitigate the mismatch between the surrogate and the real quantizer.

   Key Parameters (§5.1, §D.3)

   • Attack configuration: The attacker injects one outlier per 32 weights; user quantization group size is 128 (when configurable); all target loss weights λ = 0.05.
   • Hyperparameters (Table 7): Llama3.1/Qwen2.5 use c = 512, learning rate 2.0×10⁻⁵, no noise; Mistral uses c = 256, noise σ = 1.0×10⁻³, p = 0.75, learning rate 5.0×10⁻⁶; the switch layer is chosen from layers 19–27 depending on the scenario.
   • Training amount: kick-start fine-tuning 2 epochs, refinement fine-tuning 4 epochs, effective batch size 32; the authors state the whole pipeline takes about 2 hours.

   Success Criteria (§5.1, §D.4)

   • Jailbreak: GPT-4.1-mini scores harmfulness on a 1–5 scale, ≥ 4 counted as success.
   • Over-refusal: GPT-4.1-mini outputs one of four labels A/B/C/D, with A and B (refusal) counted as success.
   • Content injection: case-insensitive substring matching of the target word "McDonald".

   Collapse this answer

4. 04What experiments did the paper conduct?

   3 models × 3 scenarios × 12 quantization methods: after quantization the ASR of most combinations exceeds 90%, with some combinations notably lower on AutoRound and IQ4_XS.

   Experimental Setup

   • Models under test: Llama3.1-8B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3.
   • Quantization methods: zero-shot (LLM.int8(), NF4, FP4, GGUF Q4_0), DIOB (GGUF Q4_K_M, HQQ, SINQ), DDOB (GPTQ 4-bit/8-bit, AWQ, AutoRound, GGUF IQ4_XS); mainly 4-bit, calibration data for main experiments is C4.
   • Attack scenarios and data: jailbreak trained on LLM-LAT (4.9k entries), evaluated on HEx-PHI (300 entries); over-refusal and content injection trained on the modified GPT4-LLM provided by [23] (5.2k entries), evaluated on Dolly (1.5k entries).
   • Evaluation method: jailbreak and over-refusal judged by GPT-4.1-mini (jailbreak scored on a 5-point scale, ≥ 4 counted as success; refusal A/B labels counted as success); content injection uses substring matching.
   • Utility evaluation: MMLU, ARC-Challenge, HellaSwag, HumanEval, GSM8K, using Eleuther LM Evaluation Harness, reporting the score ratio of the attacked model relative to the original model.
   • Sample size and repetitions: jailbreak N = 300, others N = 1500, 1 response generated per prompt (max 512 tokens); the paper gives a 95% confidence interval formula, stating the widest is about ±5.7% (at N = 300, ASR = 50%); no multiple repetitions reported.

   Main Results

   According to Table 1 (ASR%, each attack model evaluated on all quantizers):

   Swipe horizontally to read the table

   Quantization method	Jailbreak Qwen2.5	Jailbreak Llama3.1	Jailbreak Mistral	Refusal Qwen2.5	Refusal Llama3.1	Refusal Mistral	Injection Qwen2.5	Injection Llama3.1	Injection Mistral
   BF16 attack model	6.3	2.3	28.7	0.3	1.2	4.3	1.9	2.5	4.2
   LLM.int8()	94.3	95.7	88.0	95.0	89.1	48.3	95.5	95.8	83.6
   NF4	94.7	95.7	90.0	95.8	73.7	47.7	96.5	96.8	81.3
   FP4	93.0	95.0	67.3	93.4	84.4	27.7	98.7	98.9	35.5
   GGUF Q4_0	92.0	90.0	30.7	98.3	70.4	4.3	97.9	95.7	7.3
   GGUF Q4_K_M	91.3	91.0	89.7	92.9	81.9	13.5	94.3	95.5	7.4
   HQQ	93.3	94.3	90.7	100.0	92.9	33.7	98.9	98.3	85.0
   SINQ	95.0	94.7	87.3	99.9	89.6	43.0	99.5	97.7	85.0
   GPTQ 8-bit	80.3	92.7	87.0	78.3	66.9	25.1	95.9	94.9	82.7
   GPTQ 4-bit	94.7	95.7	89.7	94.3	68.8	48.7	98.3	96.3	81.8
   AWQ	93.7	95.0	87.3	99.5	74.3	49.6	98.5	96.3	79.8
   AutoRound	50.0	21.0	86.3	37.9	10.6	41.7	90.3	19.4	64.7
   GGUF IQ4_XS	13.7	91.3	44.3	7.7	85.1	5.3	42.9	95.6	5.5

   • Authors' interpretation: The authors state that after quantization ASR is generally above 90%, while at full precision it is usually below 5%, and note that HQQ, SINQ, and all DDOB methods had never previously been successfully attacked.
   • Exceptions (per the table): Mistral's ASR is clearly lower than the other two models on most methods (e.g., GGUF Q4_0 jailbreak 30.7%, refusal 4.3%); on AutoRound, several settings for Qwen2.5 and Llama3.1 are below 51%; on GGUF IQ4_XS, several settings for Qwen2.5 and Mistral are below 45%; the paper does not explain the reasons for these low combinations.
   • The authors state: In the jailbreak scenario, the full-precision versions of the attack models (6.3/2.3/28.7) appear safer than the original models (12.0/8.0/33.7), making them more attractive to users.

   Utility Preservation

   • Relative utility (Table 2, mean attacked/original of the five benchmark scores): Qwen2.5 across the three scenarios is 98.8±1.7, 96.6±3.5, 96.2±3.5; Llama3.1 is 95.9±2.7, 94.4±5.3, 96.0±4.6; Mistral is 93.8±13.4, 81.2±17.5, 86.0±13.9.
   • Authors' interpretation: The authors state that the attack models preserve utility well, with Llama3.1 and Qwen2.5 averaging above 90% and Mistral above 80%; and state it can be further improved by adjusting the KL regularization strength, but this paper does not do model-specific tuning, fixing λ = 0.05.

   Defense Evaluation (§5.3)

   • Gaussian noise defense (Figure 2, Qwen2.5-7B jailbreak): The authors report finding no noise level that both lowers ASR and preserves utility—starting at σ = 10⁻³ utility has already dropped while the attack remains effective, and at σ = 5·10⁻³ the ASR drop is accompanied by complete utility collapse; the authors explain that prior attacks relied on tightly constrained weight ranges, whereas this attack is driven by large outliers and is unaffected by small noise.
   • Outlier detection and rescaling (Table 3, Llama3.1-8B jailbreak): At attack scale c = 512, both rescaling and zeroing reduce GPTQ 4-bit ASR from 95.7% to 1.7%/1.0%; at c = 4096 both defenses fail (GPTQ 4-bit ASR 47.0% after rescaling), and the pre-quantization ASR is raised to 43.7–45.0%. The authors believe the results indicate that a model in which such anomalies are detected should be treated as untrusted rather than repaired by rescaling.
   • Statistical detection (Figure 6, Appendix C.2): Using the two-sample KS D statistic to compare the weight distributions of the original and attack models layer by layer, the attacked layer (layer 25) shows a spike clearly higher than the ordinary fine-tuning baseline; the authors believe the attack can be found by a simple statistical test.

   Other Ablations and Analysis

   • Pipeline ablation (Table 4, Llama3.1-8B): Removing zero initialization makes utility zero (HumanEval 0.0); removing kick-start or outlier injection makes the attack fail (GPTQ 4-bit ASR 4.0%, 1.3% respectively); removing the refinement fine-tuning still leaves the attack successful (97.3%), but content injection 8-bit ASR rises from 66.9% to 95.0%, a one-tier increase (the paper states this as an improvement of 71.3%).
   • Outlier magnitude (Figure 3, Llama3.1-8B jailbreak): GPTQ 4-bit ASR rises starting at c = 2⁴ and approaches full marks at c = 2⁶; GPTQ 8-bit requires c ≥ 2⁸; utility declines starting at c = 2¹³; the authors accordingly choose c between 2⁸ and 2¹³.
   • Group size (Figure 4, Llama3.1-8B jailbreak): The attack succeeds when the attacker group size ≤ user group size (e.g., ASR 96.3% at attacker 32/user 128), and fails otherwise (2.3% at attacker 128/user 32).
   • Weight matrix selection (Table 5, Llama3.1-8B jailbreak, layer 25, c = 2⁹): FFN up_proj (GPTQ 4-bit/8-bit ASR 95.7/95.7) and gate_proj (90.7/95.0) are reliable; on the attention matrices and down_proj the 8-bit attack largely fails (ASR 4.0–7.3%).
   • Layer selection (Figure 5): Choosing the first few layers (0 or 1) reduces utility; choosing the last few layers (28–31) reduces ASR, especially for GPTQ 8-bit; middle layers attack successfully with utility preserved.
   • Calibration data (Table 6, Llama3.1-8B jailbreak): Under C4, OpenWebText, The Pile, WikiText, AWQ ASR is 93.0–95.0% and GPTQ 4-bit 94.3–96.7%, which the authors state is consistent with the intuition that the attack does not depend on calibration data.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list limitations in Appendix A in three areas: model scale, quantization method coverage, and defense research; the experiments cover only 3 models of 7B–8B scale.

   Limitations and future directions noted by the authors

   • Model scale: Due to resource constraints, larger LLMs such as 70B parameters are not covered (Appendix A).
   • Scope of quantization methods: The experiments focus on quantization schemes widely adopted by mainstream inference frameworks such as Hugging Face Transformers and vLLM, and do not include methods requiring specialized implementations or hardware support (Appendix A).
   • Insufficient defense research: The authors state that the existing Gaussian noise defense remains insufficient and call for more comprehensive research on defenses against quantization-conditioned attacks, including algorithmic-level protections and deployment-time verification mechanisms (Appendix A).
   • Weight selection analysis: The authors state that through different hyperparameter optimization, selecting other weight matrices may also achieve similar attack effects, and leave a more comprehensive weight selection analysis to future work (Appendix C.1).

   Experimental coverage

   • Models tested: Llama3.1-8B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, 3 in total, all of 7B–8B scale (§5.1).
   • Quantization methods: 12, covering three categories: zero-shot, DIOB, and DDOB; bit widths are mainly 4-bit, plus GPTQ 8-bit (§5.1).
   • Attack scenarios: 3 — jailbreak, over-refusal, and content injection; evaluation sets are HEx-PHI (300 items) and Dolly (1.5k items) (§5.1).
   • Defense evaluation: Gaussian noise defense, outlier zeroing and rescaling (assuming the injected matrix and scaling factor c are known), KS statistic detection (§5.3, Appendix C.2).
   • Not reported in the paper: the number of repetitions per experiment (only 1 response is generated per prompt, §D.4), and the agreement between judge-model verdicts and human judgments.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors state that the outlier injection attack generally achieves high ASR across 12 quantization methods, extending the threat of quantization-conditioned attacks to mainstream optimization-based methods.

   A paper on quantization-conditioned attacks: the attacker releases a model that appears benign at full precision, and the model exhibits malicious behavior after the user quantizes it themselves.

   • Problem: Existing attacks of this kind cover only zero-shot quantization methods such as FP4 and NF4 and have consistently failed against optimization-based methods such as GPTQ and AWQ, which the authors believe leads users to mistakenly assume that choosing more modern quantization methods makes them immune.
   • Method: Exploiting the shared property of quantization methods — "grouping + shared scaling within a group" — inject a large outlier into each quantization group of selected FFN matrices so that the remaining weights in the same group become zero after quantization; combined with a four-step procedure (zero initialization, dual-behavior kick-start fine-tuning, outlier injection, quantization-proxy refinement fine-tuning) to train a behavior switch before and after quantization.
   • Main results (Table 1, 3 models of 7B–8B scale × 3 scenarios × 12 quantization methods): after quantization the ASR of most combinations exceeds 90% (e.g., Llama3.1 jailbreak GPTQ 4-bit 95.7%), while at full precision it is usually below 5%; exceptions are several Mistral combinations and some combinations on AutoRound and GGUF IQ4_XS, which are notably lower.
   • Utility (Table 2): the relative utility of the attack models averages above 94% for Qwen2.5 and Llama3.1, and above 81% for Mistral.
   • Defenses: The authors report that the Gaussian noise defense is ineffective against this attack; rescaling when the scaling factor is known can defend against attacks with c = 512 but fails for c = 4096; attacked layers can be detected by the KS statistic.
   • Authors' conclusion: The authors state that quantization security risks are not limited to simple schemes but widely exist in complex, commonly used quantization methods, and call on the community to pay attention to defense research against deployment-time attacks.

   Collapse this answer

Related papers