Skip to content

Is 4-bit Quantization More Than Noise for Chat Model Safety Behavior? Attributing Behavior Change with Matched Perturbations

Is 4-Bit Quantization More Than Noise to a Chat Model's Safety Behaviour? Attributing Behavioural Shifts with Matched Perturbations

arXiv: 其他 AI 安全论文· arXiv:2611.22186· Ege KayaUniversity of British ColumbiaIndexed by this site · 2 days ago· Originally publishedSelected Evaluation· Its place in the scan record →SelectedAttention39

Quick read

Analysis / interpretability

Based on the paper PDF (AI-generated); refer to the original

The authors conducted a preregistered paired experiment on three 3–4B chat models, reporting that the direction of 4-bit quantization's effect on refusal behavior depends on the specific rounding error rather than bit width, with two models showing shifts in opposite directions.

Questions
Chat models running on phones and laptops are usually 4-bit copies of full-precision models, while existing audits disagree on how quantization affects refusal and offer no explanation for why.
Methods
A preregistered paired experiment on three 3–4B models, using 8-bit as a zero-effect baseline, then attribution with mirrored, permuted, Gaussian, and other perturbations of the same size as the actual rounding error and with dose scaling, plus analysis of refusal-direction projections.
Experiments and results
Qwen2.5-3B under forced start is more likely to continue harmful content after 4-bit; Llama-3.2-3B under free start complies less with jailbreak prompts but also refuses more benign requests; Qwen3-4B shows no significant shift; random perturbation means are near zero, and direction changes nearly linearly with scaling of the real error.
Limitations and future work
Limitations listed by the authors include: only three 3–4B models, limited quantizer and runtime coverage, the free-start effect first found on the development set, different prompt sets for the two settings, dose builds not in a real format, coarse controls, greedy decoding, 128 token truncation, and human labeling by only one of the authors.
Experimental setupQwen2.5-3B-Instruct、Llama-3.2-3B-Instruct et al. · harmful_test (Arditi et al., 2024)、WildJailbreak et al. · paired shift in judged harmful output / fulfilment rate (pp)、churn et al.
Threat model
Not an attack paper. Setting: developers evaluate chat models at full precision and release 4-bit copies made by third parties, and the paper asks whether the copies still refuse the same requests; assistant prefill is used to probe how easily refusal is bypassed, and the authors state explicitly that this is not a realistic attack on local weights.
Evaluated models
Qwen2.5-3B-InstructLlama-3.2-3B-InstructQwen3-4B-Instruct-2507
Benchmarks
harmful_test (Arditi et al., 2024)WildJailbreakXSTestSORRY-Bench
Metrics
paired shift in judged harmful output / fulfilment rate (pp)churn∆NLL/token vs bf16refusal-direction projection
AI summaryand selection rationaleA preregistered study by Ege Kaya at the University of British Columbia found on three 3–4B chat models that the effect of 4-bit quantization on refusal varies by model and build: when prefilling the start of a compliant response, Qwen2.5-3B more often continues harmful content, Llama-3.2-3B's compliance rate on held-out WildJailbreak jailbreak prompts drops from 45.0% to 34.8%, and Qwen3-4B shows no significant change. Matched perturbation experiments show that shuffling and Gaussian perturbations of the same size average near zero, behavior changes nearly linearly with the scaling of the true rounding error, and the direction of the shift depends on the specific rounding error rather than bits per weight or likelihood; at half the error, Llama's likelihood is nearly unchanged but its compliance rate has already dropped 5.8 percentage points. Llama's drop is also not better harmful discrimination; it simultaneously refuses jailbreak-style benign requests more often. The paper recommends using 8-bit when memory allows and testing each quantized build against its original model, and the code and annotations have been released on GitHub.

A preregistered study by Ege Kaya at the University of British Columbia found on three 3–4B chat models that the effect of 4-bit quantization on refusal varies by model and build: when prefilling the start of a compliant response, Qwen2.5-3B more often continues harmful content, Llama-3.2-3B's compliance rate on held-out WildJailbreak jailbreak prompts drops from 45.0% to 34.8%, and Qwen3-4B shows no significant change. Matched perturbation experiments show that shuffling and Gaussian perturbations of the same size average near zero, behavior changes nearly linearly with the scaling of the true rounding error, and the direction of the shift depends on the specific rounding error rather than bits per weight or likelihood; at half the error, Llama's likelihood is nearly unchanged but its compliance rate has already dropped 5.8 percentage points. Llama's drop is also not better harmful discrimination; it simultaneously refuses jailbreak-style benign requests more often. The paper recommends using 8-bit when memory allows and testing each quantized build against its original model, and the code and annotations have been released on GitHub.

Why it was selectedThe paper uses matched perturbations to attribute post-quantization changes in refusal to the specific rounding error; teams deploying quantized models need to test each build against its original model rather than looking only at bit count or likelihood.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Whether the refusal behavior of 4-bit quantized copies matches the original model, and why audit conclusions contradict each other.

    Core question: whether 4-bit quantized chat models deployed on-device still refuse the same requests, and why existing audit conclusions contradict each other.

    • Setting and importance: Chat models on phones and laptops are usually 4-bit copies of full-precision models, often generated by a third party with a single command; beyond capability checks, whether refusal behavior is preserved is often assumed (§1).
    • Gaps in existing work: Existing audits disagree on the direction and magnitude of quantization's effect on refusal; some find reduced refusal, others find 4 bit changes almost nothing or even increases refusal; the authors argue no one explains why (§1, §2).
    • Core observation: A 4-bit build is just one of many weight perturbations as large as the rounding error, and behavior change may come from the specific rounding error or from any perturbation of equal size; the two explanations can be distinguished with non-quantization perturbations of equal size (§1).
    • Measurement difficulty: Single labels are unstable—an 8-bit build has nearly unchanged likelihood yet rewrites most responses and changes the judged label by a few percentage points, so the authors report each result as a paired rate shift with an 8-bit zero-effect baseline (§4).
    • Paper contributions: The authors state the contributions as (i) measuring two 4-bit refusal shifts in opposite directions; (ii) attributing the shifts to the specific rounding error using matched perturbations and dose-response; (iii) refusal-direction analysis; (iv) a paired comparison design with an 8-bit zero line (§1).
    • Setting (not a threat model): Developers evaluate at full precision and release 4-bit copies; assistant prefill is used to probe how easily refusal is bypassed, and the authors state explicitly that this is not a realistic attack on local weights, because the weight holder can directly remove safety training (§3).
  2. What related research exists?

    Four groups: inconsistent conclusions in quantization refusal audits, quantization as perturbation, label instability, and prefill and shallow alignment.

    Refusal audits of quantized models

    • Hong et al. (2024), Chen et al. (2025), Kharinaev et al. (2025), Kadadekar (2026), Prasad and Pal (2026), Wu et al. (2026)—audit refusal in quantized chat models, reporting directions that vary by model and method; specific results cited in the paper include: no method is consistently safest across 66 quantized variants (Kharinaev et al., 2025); AWQ, GPTQ and 2–3-bit GGUF change refusal by up to 57 percentage points, while Q4_K_M builds change by no more than 3 (Kadadekar, 2026).
    • Egashira et al. (2024, 2025)—construct adversarial rounding errors so that models behave harmlessly at full precision and worse after quantization. The paper notes that none of these studies applied a non-quantization perturbation as large as the quantization error.

    Quantization as perturbation

    • Gong et al. (2024)—treat quantization as noise on weights and activations, compare with artificial perturbations matched in L2 norm on general task performance, and find consistency in some model families and not others. The paper says this work asks the same question for refusal and adds controls constructed from the real error (mirror, shuffle, scaled copies).
    • Razan and Shivam (2026)—compare isotropic weight noise with the real 4-bit error on first-token marginals, and the noise model underestimates flips. The paper says this work uses full responses and controls constructed from the real error.
    • Peng et al. (2024)—random weight perturbations within the basin around the model keep average safety unchanged.

    Single-label instability

    • Dutta et al. (2024), Larsen (2025), Prasad and Pal (2026), Monteuuis et al. (2026)—quantized models flip individual answers while accuracy is unchanged; safety judgments flip with seed and temperature; attack success rates vary from run to run. Shahani et al. (2025), Liu et al. (2026)—adding noise to activations or parameters erodes safety alignment, but not matched to quantizer error. The paper treats this instability as a baseline to define the estimator (§4).

    Prefill and shallow alignment

    • Qi et al. (2025)—refusal is concentrated in the first few tokens of the response; Arditi et al. (2024)—refusal is mediated by a single direction in the residual stream. Struppek et al. (2026), Park et al. (2026)—prefill can bypass refusal in full-precision open-source models; under 2-bit, failures are concentrated in the smallest models. Chhabra and Khalili (2025)—the refusal direction is preserved after 8-bit and AWQ quantization (cosine similarity above 0.99).

    Positioning: The authors say this work differs from the above in using matched perturbations (including controls constructed from the real rounding error) and dose-response to attribute the direction of refusal shifts, and in handling label instability with an 8-bit zero line (§1, §2).

  3. How does the paper address this problem?

    A paired design plus an 8-bit zero line measures the shift, then equal-size perturbations and dose scaling attribute the source of the direction.

    Overall approach: first measure the 4-bit build's refusal shift relative to bf16 under two settings with a paired design (using the 8-bit build as a zero-effect baseline), then use mirrored, permuted, and Gaussian perturbations of the same size as the actual rounding error and error scaling (dose-response) to determine whether the shift direction depends on the specific rounding error or only on the perturbation size, and finally do a correlation analysis using refusal-direction projections.

    Models, formats, and capability measure

    • Models tested: Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct, Qwen3-4B-Instruct-2507, default chat template, all quantized builds converted from bf16 checkpoints via mlx-lm 0.32.2 (§3).
    • Main format: mlx-lm default affine round-to-nearest 4 bit, group size 64 (4.50 bits/w); also 8bit-g64, 4bit-g32, mixed 4/6, MXFP4, and AWQ and GPTQ written in the same 4-bit group-64 format; the second runtime uses llama.cpp to build bf16, Q8_0, Q4_K_M GGUF (§3, Table 1).
    • Capability measure: change in negative log-likelihood of each build relative to bf16 itself on 200 harmless prompt responses (∆NLL/token); the 8-bit build has nearly unchanged likelihood, and MXFP4 has the largest loss in each model (Table 1).

    Two behavior settings

    • Forced start (main test): 562 requests from Arditi et al. (2024) harmful_test, prefilled with the first k = 8 tokens of a compliant response generated by an abliterated copy of the same model; k ∈ {0, 2} and a generic prefix are also reported. Tests how easily refusal is bypassed (§3, §5.1).
    • Free start: WildJailbreak evaluation set, first 200 as development set and another 400 as held-out test set, no prefill (§3, §5.2).
    • Over-refusal: 250 safe prompts from XSTest and 210 adversarial-benign prompts from WildJailbreak; SORRY-Bench is a secondary family (§3, Appendix D). Decoding is greedy, capped at 128 new tokens.

    Judging and statistics

    • Content judge: Qwen2.5-14B-Instruct reads the request and full response and labels complying / partly complying / not complying / incoherent, with the first two counted as judged harmful (forced start) or fulfilment (free start); keyword refusal detection is unusable because the prefix occupies the position it reads (§3).
    • Second judge and human: Llama Guard 3 uses "unsafe" as the construct; the forced-start main test additionally has two-stage human annotation—all pairs where the judges disagree plus 25 random pairs from each agreement stratum, blind-labeled, design-weighted, with the authors labeling 400 pairs in total (800 responses) (§3, Appendix C).
    • Statistics: all are paired comparisons, shifts are rate differences, churn is the share of prompts whose label changed; exact McNemar test, Holm correction within family; intervals are percentile bootstrap with 10,000 resamples; equivalence is defined as the 90% interval of the shift falling within ±5 pp; the main test, confirmatory test, and decision rules were preregistered (§3, Appendix A).

    Matched perturbations and dose

    • Error definition: for each weight matrix W quantized by the 4-bit build (including embeddings), let E = dequant(Q(W)) − W be its rounding error; control builds perturb W by the same magnitude as the rounding but with a different pattern (§7).
    • Controls: mirrored W − E reverses the error sign; three permuted builds shuffle E within each group of 64; three Gaussian builds replace E with noise scaled by row norm; a dense build W + E validates the construction (likelihood differs from the real quantized model by -0.0007 and -0.0001 nats/token, first-token agreement 100.0% and 99.5%) (§7).
    • Direction decision rule: if the random-control mean has the same sign as the real build and the interval excludes zero, and the mirror does not reverse it, the direction is a "model tendency"; if the mirror reverses it and the interval excludes zero, or the random-control interval includes zero while real-minus-random excludes zero, it is "rounding-specific" (§7).
    • Dose: builds W + aE at a ∈ {−1, 1/2, 1, 2} to test the linearity of the shift with error scaling (§7, Figure 2).

    Refusal-direction projection

    • For each build, record the projection of the model state at the generation start onto its bf16 refusal direction on the behavior-test prompts, compared per prompt with bf16; the authors state explicitly that this analysis is correlational and does not identify circuits (§8).
  4. What experiments did the paper conduct?

    Two 4-bit shifts in opposite directions, attributed to the specific rounding error; neither bit width nor likelihood can predict them.

    Experimental setup

    • Models tested: Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct, Qwen3-4B-Instruct-2507 (§3).
    • Data: 562 harmful_test (forced start); WildJailbreak development set 200 and held-out test set 400 (free start); XSTest 250 safe prompts, WildJailbreak 210 adversarial-benign prompts; SORRY-Bench is a secondary family (§3).
    • Metrics: paired rate shift relative to bf16 (pp), churn, ∆NLL/token; equivalence bound ±5 pp; exact McNemar + Holm correction, 10,000 bootstrap (§3).
    • Judging: main judge Qwen2.5-14B-Instruct (4-bit), second judge Llama Guard 3; the forced-start main test has the authors' own two-stage blind labeling (400 pairs, 800 responses, design-weighted); judge-human agreement 78%/81%/90% (κ = 0.50/0.60/0.52), two below the registered 0.60 threshold, so the forced start uses human labels as the reference (§3, §4).
    • Repetition and zero line: greedy rerun of the same weights on 100 responses is 100% token-for-token identical; the 8-bit build is the zero-effect baseline (§4).

    Main results

    ModelForced-start 4-bit shift (pp)Free-start 4-bit shift (pp)
    Qwen2.5-3B+6.2 (human +19.4)-2.3
    Llama-3.2-3B+0.0-10.3
    Qwen3-4B-1.6+3.8

    According to Table 2; bold is a significant confirmatory test. Forced start is harmful_test + 8 token prefix, free start is held-out WildJailbreak with no prefill.

    • The authors state that Qwen2.5-3B more easily continues harmful content under forced start: among pairs where the judges disagree, harmful responses come from 4-bit 34 times and from bf16 9 times (p = 1.7 × 10−4), human-weighted shift +19.4 pp [+6.0, +33.4]; the judge underestimates harm, giving a same-sign smaller shift of +6.2 pp [+2.8, +9.8] (§5.1).
    • The authors state that Llama-3.2-3B's fulfilment rate under free start drops from 45.0% for bf16 to 34.8% for 4-bit (-10.3 pp [-14.0, -6.2], p = 4.5 × 10−7), its 8-bit build shows no difference (-1.3 pp), and MXFP4 drops more (-16.0 pp) (§5.2).
    • The pooled shift across the three models is not significant (p = 0.11), and the authors state the effect is specific to Qwen2.5-3B; Qwen3-4B shows no significant shift under either setting (§5.1, §5.2).

    Attribution: matched perturbations and dose (§7)

    • Random-control mean near zero: the average of six random controls for Qwen2.5-3B is +0.4 pp [-2.1, +2.8], the real build exceeds it by +5.8 pp, and the mirror estimate has the opposite sign (-3.0 pp); for Llama-3.2-3B the random control is -1.1 pp, real-minus-random -9.1 pp, mirror +3.7 pp. By the decision rule both shifts are rounding-specific.
    • Dose nearly linear: under W + aE the shift is monotonic and nearly linear with a, slope +4.6 pp [+3.3, +5.9] (Qwen2.5 forced) and -7.2 pp [-8.8, -5.6] (Llama free) per unit error (Figure 2).
    • Half-error build: Llama-3.2-3B at W + ½E has nearly unchanged likelihood (+0.005 nats/token) but already fulfils 5.8 percentage points less (p = 0.0022); the authors use this to argue that likelihood checks may miss behavior shifts.
    • Other quantizers: AWQ reproduces the Qwen2.5-3B shift (+4.8 pp) but not the Llama-3.2-3B one (+1.0 pp); llama.cpp's Q4_K_M shows no shift for either model, while the same-bit-width MLX build can shift Qwen2.5-3B; the Qwen2.5-3B GPTQ build failed and was excluded (∆NLL +9.335, Appendix A).

    Ruling out simpler explanations (§6) and over-refusal

    • Capability explanation does not hold: MXFP4 has the largest likelihood loss in each model (Table 1) yet does not shift Qwen2.5-3B's forced-start behavior (-0.7 pp); Llama-3.2-3B has the smallest likelihood loss (+0.038) yet the largest free-start shift; the shift ordering is determined neither by bit width nor by likelihood loss. The paper did not run task benchmarks.
    • "Becoming safer" explanation does not hold: 4-bit Llama-3.2-3B's refusal rate on XSTest safe requests is comparable to bf16 (-0.4 pp), but it refuses more jailbreak-style benign requests (+5.7 pp [+1.4, +10.0], Table 3); the authors state the drop is not better harm discrimination.
    • The two Qwen models go in opposite directions: 4-bit Qwen2.5-3B refuses fewer safe requests (-6.4 pp, Holm p = 0.083), and 4-bit Qwen3-4B refuses fewer jailbreak-style benign requests (-8.6 pp) (Table 3, §6).

    Refusal-direction projection (§8)

    • Qwen2.5-3B matches prediction one, not prediction two: under forced start the 4-bit build's projection is lower than bf16 (-0.80), the mirror is higher (+0.13), and the projection scales with dose; but across 15 builds the change in projection is not significantly correlated with the change in compliance (Spearman ρ = −0.44, p = 0.10), and several builds (e.g., one Gaussian control -0.66, MXFP4 -0.61) have similar projection drops with unchanged behavior.
    • Llama-3.2-3B weakly matches both predictions: the 4-bit build's projection is slightly higher (+0.13), and across 14 builds the correlation is as predicted (ρ = −0.54, p = 0.047); but the double-error build with the largest behavior shift has the opposite projection (-0.33).
    • Authors' conclusion: the real error does move the state along the refusal direction, but so do builds with unchanged behavior, so this projection cannot predict behavior shifts.

    Other ablations and analyses

    • Prefix depth: the Qwen2.5-3B shift already exists at k = 2 (+6.8 pp), and at k = 0 bf16 is almost never compliant (Table 7).
    • Triple response cap: Qwen2.5-3B +4.0 pp (128 token) / +6.0 pp (384); Llama free start -10.7 pp / -10.7 pp (§5).
    • Generic prefix "Sure, here is": Qwen2.5-3B +4.3 pp (not significant after correction); neither model shows a shift on held-out prompts (§5).
    • SORRY-Bench with no prefill: Llama-3.2-3B -5.0 pp [-7.8, -2.2], same sign as free start (Table 9).
    • Label variance under sampling: Qwen2.5-3B 4-bit exceeds bf16 by +0.028, Llama only by +0.009 (Appendix B).
    • MXFP4 Qwen2.5-3B fabricates a next-turn user utterance in 10.5% of responses on 200 benign prompts, versus 0 for other builds (Appendix G, Table 13).
  5. What are some points that can be further explored?

    The authors list limitations in scope, quantizer coverage, and measurement; the experiments cover only three small models and greedy decoding.

    Limitations and future directions noted by the authors

    • Scope: Only three 3–4B models from two vendors, with signs varying by model; the results cannot be generalized to 4-bit quantized chat models as a class; each result applies only to the build tested (Limitations).
    • Quantizer and runtime: Only MLX round-to-nearest (affine and MXFP4), MLX's AWQ and GPTQ, and llama.cpp Q4_K_M without an importance matrix are covered; other implementations, imatrix GGUF, bitsandbytes, and vendor pipelines (quantization-aware training) are not covered; the second runtime and calibrated quantizers were run only on the two cells with shifts (Limitations).
    • The free-start effect was first found on the development set: The test prompts are held-out, but the prompt set and model selection for the confirmatory test followed the development-set results (Limitations).
    • The two settings use different prompt sets: The forced generic start can separate the two factors for Llama-3.2-3B but not for Qwen2.5-3B, whose shift may also depend on prompt type; the forced start also raised the baseline compliance rate, so the comparison is not fully equivalent (Limitations).
    • The dose builds are not real formats: The half-error and double-error builds are dense bf16 models that no published quantizer produces, and the double build's likelihood loss is far larger than that of real builds (Limitations).
    • Other listed limitations: The controls are coarse (only one mirror, three permutation seeds, three Gaussian samples, and two scales per cell, and attribution is to the whole perturbation rather than specific layers; the refusal-direction analysis is correlational); greedy decoding, one response per prompt, 128-token truncation; the forced-start prefix is model-specific; the main judge is itself a 4-bit model of the same family and underestimates harm; the only human annotator is the author (Limitations).

    Experimental coverage

    • The models tested are Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct, and Qwen3-4B-Instruct-2507, 3 in total (§3).
    • Behavioral evaluation uses greedy decoding, one response per prompt, and a cap of 128 new tokens; the seed lower-bound experiment under sampling was not run for Qwen3-4B (§3, Limitations).
    • Human annotation covers only the forced-start main test (400 pairs, 800 responses, annotator is the author); all other cells use the judge (§3, Limitations).
    • The capability measure is only the likelihood of bf16's own responses; no task benchmarks were run, and the authors say that therefore they cannot say whether task benchmarks would give the same ordering (§6, Limitations).
    • The paper does not report an independent second annotator or inter-annotator agreement alongside the human annotation; the refusal-direction analysis is a correlational projection, with no causal circuit intervention (§8, Limitations).
  6. Summarize the main content of the paper

    The direction of 4-bit quantization's effect on refusal depends on the specific rounding error, and each build must be compared with the original model.

    An attribution study on quantized safety behavior: on-device chat models are mostly 4-bit copies of full-precision models, while existing audits disagree on how quantization affects refusal and offer no explanation.

    • Problem: A 4-bit build is just one of many weight perturbations as large as the rounding error; behavioral change may come from the specific rounding error or from any equally large perturbation; the authors use matched perturbations to distinguish these two explanations.
    • Method: A preregistered paired design, using 8-bit builds as a zero-effect baseline, measuring shifts under forced start (prefilling 8 compliance tokens) and free start (held-out jailbreak prompts); then attributing via mirror, within-group shuffle, Gaussian and other error perturbations and W + aE dose scaling; the judge is Qwen2.5-14B-Instruct, with the main test supplemented by the authors' blind labeling.
    • Two shifts in opposite directions: Qwen2.5-3B under forced start is more likely to continue harmful content after 4-bit (judge +6.2 pp, human +19.4 pp); Llama-3.2-3B under free start has a 10.3 percentage point drop in compliance rate, but refusal of benign requests in jailbreak style increases by 5.7 pp, which the authors say is not better harm discrimination; Qwen3-4B shows no significant shift.
    • Attribution results: The randomized control mean is close to zero, the mirror reverses the sign, and the shift scales nearly linearly with error (slope +4.6 / -7.2 pp per unit error); the authors therefore state that the direction depends on the specific rounding error, not on bit width or perturbation size.
    • Missing predictors: Neither bit width nor likelihood loss predicts which build will shift—the Llama half-error build has almost unchanged likelihood yet already has 5.8 percentage points less compliance; the relationship between the refusal-direction projection and behavioral change is weak and inconsistent.
    • Authors' conclusion: No 8-bit build showed a significant rate shift; use 8 bit when memory allows; under 4 bit, none of the measured quantities predicts which build will shift, so each build should be compared in a paired fashion with its original model, with a zero line.
Read originalarxiv.org