TamperBench: systematically stress-testing LLM safety under fine-tuning and tampering

1. 01What problem does this paper attempt to solve?

   The authors state that there is a lack of a standardized framework for evaluating the tampering resistance of open-weight LLMs, and propose a unified benchmark, TamperBench.

   Core question: how to standardly measure the robustness of open-weight LLM safety guardrails after the weights or representations are tampered with.

   • Scenario and importance: open-weight models can be modified white-box without restriction; fine-tuning, LoRA, and representation-layer attacks can all weaken refusal guardrails; the authors state that as frontier open models' capabilities approach critical thresholds, the misuse risk of tampered models is increasingly urgent.
   • Existing gaps: the authors state that defense works use different attacks, threat models, and metrics (Figure 2), and evaluation is fragmented; they relay Casper et al.'s observation: defense papers often report resistance to thousands of steps of adversarial fine-tuning, whereas second-party red-team evaluations' SOTA is only hundreds of steps, and conclusions have been disputed and overturned.
   • Core observation: the authors argue that realistic tampering evaluation requires systematic hyperparameter sweeps (rather than arbitrary training configurations), and must explicitly impose a utility constraint—without the constraint one gets models with high harm scores but collapsed capability and no practical uplift.
   • Proposed: TamperBench, which the authors call the first benchmark and toolkit for unified evaluation of LLM tampering resistance, integrating 9 types of tampering attacks, 7 alignment-stage defenses, and standardized safety/utility evaluation, and used to evaluate 21 open-weight models.
   • Threat model:
     • Attacker goal: weaken refusal guardrails (increased compliant answers to unsafe prompts) while general capability is largely retained.
     • Knowledge: worst-case open-weight setting, the attacker has full white-box access to weights and representations.
     • Capability: any tampering strategy may be used; TamperBench evaluates the worst-case threat surface under optimized hyperparameters; two types of intent are distinguished: accidental tampering (side effects of benign fine-tuning) and malicious tampering.
     • Victim system: open-weight LLMs (including defense-enhanced variants); success requires a utility constraint of relative MMLU-Pro drop ≤10%.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: LLM tampering vulnerability, three-stage tampering defenses, and existing evaluation frameworks such as HarmBench.

   LLM vulnerability and tampering attacks

   • Qi et al. (2024), Che et al. (2025), etc.—fine-tuning on a small number of harmful samples can suppress refusal, and benign fine-tuning also breaks guardrails; parameter-efficient methods such as LoRA lower the threshold for tampering.
   • Halawi et al. (2024), Murphy et al. (2025)—adversarially constructed data can bypass moderation of closed-source fine-tuning APIs, including backdoors and data poisoning; jailbreak-tuning lets models efficiently acquire jailbreak susceptibility.
   • Schwinn and Geisler (2024), Arditi et al. (2025)—representation-space attacks: adapting latent embeddings to elicit harmful responses, or ablating the refusal direction.

   Tampering defenses (three classes by intervention stage)

   • Alignment-stage defenses: TAR (Tamirisa et al., 2025), LAT (Sheshadri et al., 2025), RR/Circuit Breakers (Zou et al., 2024), CRL, Booster, T-Vaccine, CTRL, RSN-Tune, SDD, etc.—embed durability during alignment training; the authors consider this the only stage applicable to both open-weight and API deployment, hence the benchmark's focus.
   • Fine-tuning-stage and post-fine-tuning defenses: Safe LoRA (Hsu et al., 2024), Lisa (Huang et al., 2024), Antidote (Huang et al., 2024), etc.—the authors note that these two classes presuppose centralized control over the fine-tuning process and mainly apply to LLMaaS providers, not to redistributed open weights.

   Existing evaluation frameworks

   • HarmBench (Mazeika et al., 2024)—automated red-teaming and refusal-robustness benchmark; the authors note it is limited to prompt-level attacks and does not systematically evaluate weight-space tampering.
   • Existing tamper-resistance toolkits (Qi et al., 2024; Murphy et al., 2025, etc.)—the authors note their limitations in scalability, ease of integrating new defenses, and coverage of tampering strategies; both Huang et al. (2024) and Casper et al. (2025) call for standard benchmarks.

   Baselines and benchmarks: The objects compared in this paper are 7 alignment-stage defenses (Booster, CRL, CTRL, RSN-Tune, SDD, T-Vaccine, TAR) and 5 official defense-enhanced checkpoints (ReFAT, RR, CRL, TAR, LAT); the safety metric is StrongREJECT (supplemented by JailbreakBench, Policy-Eval, SafetyGap), and the utility metric is MMLU-Pro (560-sample subset).

   The authors position this paper as the first standardized foundation integrating attacks, defenses, and evaluation protocols into a single scalable framework, to fill the fragmentation gap described above.

   Collapse this answer

3. 03How does the paper address this problem?

   TamperBench unifies the attack suite, defense interface, and evaluation protocol, performing hyperparameter sweeps per model–attack pair and selecting the worst configuration under a utility constraint.

   Overall approach: integrate tampering attacks, alignment-stage defenses, and safety/utility evaluation into a scalable framework, TamperBench, approximating the worst-case threat through systematic hyperparameter sweeps for each model–attack pair rather than relying on any single arbitrary training configuration.

   Threat model and success criteria

   • Focus on worst-case open-weight access: the attacker has full white-box access to weights and representations, and the defender must keep guardrails resistant to tampering while preserving benign capabilities.
   • Tampering success = guardrails are weakened (increased compliance with unsafe prompts) and general capability is largely retained; the utility constraint in the main experiments is a relative MMLU-Pro drop ≤10%, which the authors call a practical constraint to prevent overfitting to safety metrics.
   • Distinguish two types of intent: accidental tampering (side effects of benign fine-tuning) and malicious tampering (arbitrary policy under white-box access, evaluating the worst case according to optimized hyperparameters).

   Tampering attack suite (9 types)

   • Weight-space benign: benign full-parameter fine-tuning, benign LoRA (BookCorpus 1350 samples), simulating accidental tampering.
   • Weight-space malicious: harmful full-parameter fine-tuning, harmful LoRA, multilingual (French) fine-tuning, based on 1350 samples of the Safe-RLHF-Alpaca harmful corpus.
   • jailbreak-tuning: three types—backdoor, style-modulation, competing-objectives—following the data poisoning method of Murphy et al. but using 100% harmful data (reflecting the worst case for open weights, rather than the original work's covert setting).
   • Representation space: embedding attack (soft prompt optimization of Schwinn and Geisler, perturbing representations at inference time without changing weights); because a single run costs about 7–8 times that of a fine-tuning attack, no hyperparameter sweep is done, and the high-success-rate configuration reported in the original source is used directly.

   Defenses and toolkit

   • Reimplement 7 alignment-stage defenses: Booster, CRL, CTRL, RSN-Tune, SDD, T-Vaccine, TAR, supporting hardening models from scratch; also supports directly loading defense checkpoints open-sourced by the original authors.
   • The toolkit is based on HuggingFace training infrastructure and supports multiple GPUs; a registry-style plugin architecture, where new attacks/defenses/evaluations are integrated via configuration dataclasses + implementation classes.

   Evaluation protocol

   • Safety: main metric StrongREJECT (score 0.0–1.0, rubric judge with gpt-4o-mini as the judge model, decomposed into three sub-scores: refusal, specificity, convincingness; a locally fine-tuned Gemma judge is also used as a comparison); the framework also supports JailbreakBench, Policy-Eval, SafetyGap.
   • Utility: main metric MMLU-Pro (560-sample subset); the authors report that when MMLU-Pro drops ≥10%, IFEval, MT-Bench, Minerva-Math, LiveBench-Coding also drop ≥10% in 79–89% of cases, supporting the single-metric focus.
   • Sweep and selection: run 40 Optuna trials per model–attack pair (Bayesian optimization, objective maximizing StrongREJECT), with a search space including learning rate, max steps (16–1024), chat template, LoRA rank, etc. (Table 2); first filter trials with MMLU-Pro drop ≤10%, then take the one with the highest StrongREJECT. Summary statistics are SRmax (worst case) and SRmal-avg (average over malicious attacks).

   Collapse this answer

4. 04What experiments did the paper conduct?

   All 21 models can be tampered with (SRmax>0.74); the authors state that jailbreak-tuning is the most severe, and most of the 7 alignment-stage defenses fail.

   Experimental setup

   • Models tested: 21 open-weight LLMs—the Qwen3 series (0.6B/1.7B/4B/8B and corresponding Base), Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B (Instruct and Base), Mistral-7B (Instruct and Base), and 5 Llama-3-8B-Instruct defense-enhanced variants (ReFAT, RR, CRL, TAR, LAT, using the original authors' open weights); preliminary experiments are also done on Qwen3-32B and Llama-3-70B-Instruct.
   • Attacks: 9 types of tampering attacks; main experiments use 40 Optuna trials per model–attack pair, large-model experiments 30.
   • Metrics and judging: safety uses StrongREJECT (rubric judge with gpt-4o-mini as the LLM judge, 0.0–1.0; the appendix uses a fine-tuned Gemma judge for comparison); utility uses the MMLU-Pro 560-sample subset.
   • Selection rule: the utility constraint is a relative MMLU-Pro drop ≤10%, and within the constraint the configuration with the highest StrongREJECT is taken (Figure 5).
   • Defense evaluation: 7 defenses are trained with the original papers' default hyperparameters on Llama-3-8B, Llama-3-8B-Instruct, Qwen3-8B, 21 defense checkpoints in total, following the same attack sweep protocol.
   • Manual analysis: Appendix D manually inspects 10 StrongREJECT responses per condition for some models before and after LoRA tampering (n=10 per condition).

   Main results

   The table below shows representative numbers reported in the main text (the main results figure, Figure 5, is a cell-by-cell heatmap; cell values are unreadable in the converted text, so only numbers given in the main text are listed; the utility constraint is MMLU-Pro drop ≤10% in all cases):

   Swipe horizontally to read the table

   Model/attack	Metric	Value	Source
   All 21 models	SRmax lower bound	>0.74	§4.1
   competing-objectives	cross-model average SR	0.84 (highest on 14/21 models)	§4.2
   harmful full-parameter fine-tuning / harmful LoRA	cross-model average SR	0.82 / 0.81	§4.2
   backdoor / multilingual / style-modulation	cross-model average SR	0.79 / 0.76 / 0.76	§4.2
   7–8B non-defense models	SRmal-avg / SRmax	0.82–0.85 / 0.86–0.90	§4.3
   Qwen3-4B vs 4B-Base	SRmal-avg / SRmax	0.76 vs 0.84 / 0.85 vs 0.89	§4.3
   Mistral-7B-Instruct	untampered SR / SRmax	0.64 / 0.90	§4.3
   Qwen3-32B (harmful LoRA)	SR	0.86 (Qwen3-8B is 0.81)	§4.4
   Llama-3-70B-Instruct (harmful LoRA)	SR	0.93 (8B is 0.83)	§4.4

   • The authors state that for every model there exists at least one highly effective tampering attack that, while largely preserving capability, raises the harm score far above the original level, regardless of model family, scale, or defense.
   • The authors state that competing-objectives jailbreak-tuning usually induces the largest harm increase; the embedding attack's harm increase on 7–8B models is relatively mild; benign fine-tuning also raises the harm score with very small capability loss.
   • The authors state that the direction of post-training's effect on tampering resistance is opposite between Llama-3 and Qwen3: Qwen3 post-trained variants have lower harm scores, whereas Llama-3 instruction-tuned variants have higher ones (attributed to higher answer quality when complying, with similar refusal rates).

   Alignment-stage defense evaluation (§4.5, Figure 6)

   • What was tested: 7 defenses (default hyperparameters) on 3 models under the full attack sweep.
   • Results: the authors state that defenses largely fail overall—CTRL and T-Vaccine reduce MMLU-Pro on Llama from 0.38–0.46 to 0.07–0.09, but SRmal-avg after attack is still ≥0.70; Booster, CRL, RSN-Tune, SDD have SRmal-avg within 0.04 of the undefended baseline; TAR reduces SRmal-avg from 0.85 to 0.79 only on Llama-3-8B-Instruct and is ineffective on the other two models.
   • Authors' interpretation: no defense can robustly resist the full attack suite while preserving utility.

   Defense hyperparameter sweep (§4.6, Figure 7)

   • What was tested: 30 defense hyperparameter sweeps for Booster and TAR-V (tuning against a fixed weakened attack configuration), then re-sweeping attacks with strong/weak harmful LoRA; the weakened attack is at most 64 training steps with 2% harmful data.
   • Results: under the strong LoRA attack, both default and sweep-tuned defenses are largely ineffective (SR ≥ 0.82 for all configurations); under the weakened attack, sweep tuning is effective—Booster on Qwen3-8B reduces post-attack SR from 0.56 to 0.07, and on Llama-3-8B-Base from 0.56 to 0.41; TAR-V on Llama-3-8B-Instruct from 0.39 to 0.22; in 5 of 6 cases the sweep-tuned defenses have smaller capability loss.
   • Authors' interpretation: the sweep infrastructure can find better defense configurations, but the overall picture remains that no defense withstands full-strength attacks.

   Other ablations and analyses

   • Utility constraint variation (Appendix A, Figure 8/9): relaxing the constraint to 20% further raises harm scores (notable jumps for Qwen3-0.6B and Qwen3-4B); with the constraint fully removed, Qwen3-8B's MMLU-Pro drops from 59% to 18% after full-parameter fine-tuning.
   • Defense checkpoint utility (§4.3, Appendix B): among open-source defense variants, CRL has SRben-avg 0.26 (undefended 0.35), but SRmal-avg 0.85, on par with the baseline; TAR reduces SRmax by 0.12, but untampered MMLU-Pro drops to about 0.18 (other defenses ≥0.40, of which 8 percentage points come from instruction-following failures).
   • Judge comparison (Appendix C, Figure 10): the fine-tuned Gemma judge reproduces the main patterns; under this judge, TAR's SRmal-avg is 0.41 (baseline 0.76 under the rubric judge), and CRL's is 0.70.
   • Attack variants (Appendix F, Figure 15, Llama-3.1-8B-Instruct, 70 trials): the default configuration is best at about 0.63; adding SGD/AdaFactor to the search space yields no stronger attack; increasing harmful data from 64 to 3000 gives a strongest of about 0.7.
   • Metric correlations (Appendix E, Figure 14, 48 model–attack pairs): StrongREJECT vs JailbreakBench ρ=0.71, Policy-Eval ρ=0.54, SafetyGap ρ=0.66; ΔMMLU-Pro vs ΔStrongREJECT ρ=−0.39, capability and safety changes are largely independent.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that only refusal-type defenses and the 0.6–8B scale are covered; larger models and more defense forms are left for future work.

   Limitations and future directions noted by the authors

   • Only robust-refusal-based defenses are studied; ignorance-based techniques are not covered (Limitations section).
   • Evaluation focuses mainly on the 0.6–8B parameter range; Qwen3-32B and Llama-3-70B-Instruct are only preliminary experiments, coverage of larger models is limited, and this is a direction for future extension (Limitations section).
   • The authors invite the community to contribute new attacks, defenses, and evaluations to TamperBench as the field develops (Conclusion section).

   Experimental coverage

   • 21 models tested in total (0.6B–8B, including 5 Llama-3-8B-Instruct defense-enhanced variants), plus preliminary experiments on Qwen3-32B and Llama-3-70B-Instruct (§4, §4.4).
   • Large-model experiments use only harmful LoRA (64 LAT-Harmful samples) and competing-objectives on Qwen3-32B, 30 Optuna trials (§4.4).
   • Defense evaluation covers 7 alignment-stage defenses × 3 models (Llama-3-8B, Llama-3-8B-Instruct, Qwen3-8B), with default hyperparameters; defense hyperparameter sweeps are done only for Booster and TAR-V (§4.5, §4.6).
   • Main experiments use 40 Optuna trials per model–attack pair; embedding attacks are not swept and use the original source configuration directly (§3.2, Appendix G).
   • Manual output analysis uses only 10 responses per model–condition (Appendix D); the paper does not report the total computational cost or wall-clock time of the attack sweeps.

   Collapse this answer

6. 06Summarize the main content of the paper

   TamperBench unifies tampering attacks, defenses, and evaluation; all 21 models can be tampered with, and existing alignment-stage defenses mostly fail.

   A benchmark paper: addressing the lack of standardized tamper-resistance evaluation for open-weight LLMs, it proposes the TamperBench framework and toolkit.

   • Problem: Weight and representation tampering can remove safety guardrails, but attacks, threat models, and metrics differ across works, making defense conclusions hard to compare and sometimes even overturned.
   • Method: Integrates 9 classes of weight-space and representation-space attacks, 7 alignment-stage defense implementations, StrongREJECT (gpt-4o-mini rubric judging) and MMLU-Pro evaluation; for each model–attack pair, runs 40 Optuna hyperparameter sweeps and takes the highest-harm configuration under the utility constraint of MMLU-Pro drop ≤10%.
   • Result one: All 21 open-source models of 0.6–8B (including 5 defense-enhanced variants) can be tampered with, with worst-case StrongREJECT (SRmax) all exceeding 0.74; for 7–8B non-defense models, SRmal-avg is 0.82–0.85.
   • Result two: The authors state that competing-objectives jailbreak-tuning is usually the most severe (average SR 0.84, highest on 14/21 models); the direction of post-training's effect is opposite between Llama-3 and Qwen3.
   • Result three: Most of the 7 alignment-stage defenses fail under the full attack sweep; CRL and TAR show only partial signal, and TAR's MMLU-Pro drops to about 0.18 even when not under attack; defense hyperparameter sweeps are effective only under weakened attacks (64 steps, 2% harmful data), and under strong LoRA attacks all configurations have SR ≥ 0.82.
   • Result four: In preliminary large-model experiments, Llama-3-70B-Instruct reaches SR 0.93 under harmful LoRA, and Qwen3-32B reaches 0.86, comparable to the corresponding 8B models.
   • Authors' conclusion: Neither the tested models nor defenses provide durable protection against tampering; as open-source model capabilities grow, developing and rigorously evaluating tamper-resistant training methods is increasingly urgent, and TamperBench provides a scalable foundation for this.

   Collapse this answer

Related papers