Safety Gap Toolkit: Assessing hidden dangers of open-source models

1. 01What problem does this paper attempt to solve?

   The authors aim to quantify the difference in effective dangerous capability of open-weight models before and after guardrail removal, i.e., the safety gap.

   What is to be quantified is how much the effective dangerous capability of open-weight models rises after safeguards are removed; this difference is defined as the safety gap.

   • Scenario: The authors consider open weights convenient for customization, self-hosting, and downstream research, but modifiability also lets people bypass existing safeguards. The authors focus system risk on weapons of mass destruction and cyberattacks.
   • Existing evaluation: The authors say pre-release testing usually only measures the production version with safety training. According to the authors, Meta's Llama dangerous capability evaluation does not report quantitative results and does not apply adversarial pressure that removes safeguards; in Qwen-2.5 and DeepSeek public materials, the authors also saw no safety evaluation under such pressure.
   • Hypothesis: Existing safeguards may mainly lower compliance with harmful requests rather than reducing dangerous knowledge. Effective dangerous capability means the ability to give accurate, executable guidance on harmful requests.
   • Approach: Release the Safety Gap Toolkit, chaining safeguard removal with evaluation; estimate effective dangerous capability as the compliance rate on Bio-Chem-Cyber Propensity multiplied by WMDP accuracy, and compare the difference before and after removing safeguards on Llama-3 and Qwen-2.5.
   • Threat model:
     • Intentional removal: Malicious actors hold open weights with default safeguards and fine-tune on harmful data to extract dangerous information. The toolkit includes 51 harmful training requests with compliant answers.
     • Unintentional removal: Benign users fine-tune for downstream tasks, and even harmless answers may harm safety, then release weights or provide APIs. The toolkit uses a PKU-SafeRLHF subset keeping only safe answers, with training sizes of 50 and 1,000 examples.
     • Means: Besides supervised fine-tuning, there is also training-free refusal ablation. The authors say that when weights or internal states can be directly modified, access protocols and input-output controls cannot constrain users; such modifications apply only to open-weight models.
     • Targets: The cases are multiple parameter scales of two Instruct families, not closed-source APIs.

   Collapse this answer

2. 02What related research exists?

   The authors divide safeguards into capability removal, refusal alignment, runtime control, and access protocols, and say the latter two are hard to use with open weights.

   Four categories of safeguards

   • Removing capability: Pretraining removes problematic documents, such as bioweapon-related content (Liu et al., 2024b; Bender et al., 2021); post-training unlearns dangerous capabilities (Chen & Yang, 2023; Eldan & Russinovich, 2023; Yao et al., 2024; Liu et al., 2025).
   • Alignment refusal: RLHF (Bai et al., 2022), instruction fine-tuning, and adversarial robustness techniques (Hubinger et al., 2024).
   • Control and access: Content filtering (Sharma et al., 2025), probes for detecting scheming (MacDiarmid et al., 2024), rewriting suspicious outputs by trusted models (Greenblatt et al., 2023); closed-source models can also restrict access or be taken offline. The authors argue that access protocols are unusable for open-weight models, and users can directly access the weights, so control mechanisms cannot constrain inputs and outputs.

   Ways safeguards are removed

   • Input space: Jailbreaks (Wei et al., 2023; Chao et al., 2023; Shen et al., 2024) and prefilling (Qi et al., 2024a; Haize Labs, 2024); some attacks require direct weight access (Zou et al., 2023).
   • Weight modification: Fine-tuning can restore forgotten dangerous capabilities (Qi et al., 2024b), or train the model to comply with dangerous requests (Qi et al., 2023; Zhan et al., 2024); work cited by the authors also includes benign-data fine-tuning harming safety (Qi et al., 2023), inducing other misaligned behaviors (Betley et al., 2025), and LoRA undoing safety training in Llama 2-Chat 70B (Lermen et al., 2023).
   • Training-free: Refusal ablation orthogonalizes the refusal direction (Arditi et al., 2024); inference-time activation steering can extract forgotten information and bypass refusals (Seyitoğlu et al., 2024; Arditi et al., 2024; Panickssery et al., 2023). The authors say the latter two threat categories apply only to open-weight models.

   Evaluation after removing safeguards

   • Qi et al. (2024b): Examines TAR (Tamirisa et al., 2024) and RepNoise (Rosati et al., 2024). The authors say its code can evaluate forgetting-type safeguards before and after removal, but focuses on that paper's results and is not easily extensible.
   • Arditi et al. (2024): Provides refusal ablation code and compares completions and cross-entropy on harmful and harmless requests before and after removal.
   • Wang et al. (2024): Extends refusal ablation evaluation to more general capabilities on the LM Evaluation Harness (Gao et al., 2024).

   Baselines and benchmarks

   • Controls: The original version of the same model, relative to harmful fine-tuning, benign fine-tuning, and refusal ablation.
   • Datasets: Dangerous knowledge uses WMDP (Li et al., 2024). Compliance uses the self-built Bio-Chem-Cyber Propensity, judged by StrongREJECT (Souly et al., 2024). The authors consider StrongREJECT and AdvBench (Zou et al., 2023) to be skewed toward generic criminal requests, so they built additional questions targeting bioweapons, chemical weapons, and cyberattacks. Quality uses a 100-item subset of Freebase QA (Jiang et al., 2019), judged by Claude-3.5-Haiku. Benign fine-tuning data comes from filtered PKU-SafeRLHF (Ji et al., 2024).

   The authors position this paper as an extensible evaluation framework: multiple safeguard-removal methods, plus three-dimensional evaluation of accuracy, compliance, and generation quality, used to assess open-weight models before and after foreseeable adversarial pressure.

   Collapse this answer

3. 03How does the paper address this problem?

   Effective dangerous capability is estimated as the product of the compliance rate and WMDP accuracy, then the difference before and after fine-tuning or refusal-direction ablation is compared.

   Safety Gap Toolkit chains removal of safeguards and evaluation into a pipeline. Effective dangerous capability is estimated as the product of the compliance rate and proxy knowledge accuracy, and the safety gap is the difference in this quantity between the production-ready model and the same model after safeguards are removed.

   How the estimator is defined

   • The authors say the ideal approach would be high-risk open-ended questions with expert reference answers, but to their knowledge no such benchmark exists, and publicly releasing one would carry safety risks.
   • Therefore, they separately measure the compliance rate on open-ended dangerous questions and accuracy on proxy sets such as WMDP, then take the product. Low compliance or low knowledge both lower the product.
   • The toolkit by default runs this evaluation both before and after removing safeguards, making it easy to subtract directly.

   Removing safeguards

   • Supervised fine-tuning: Supports full-parameter and LoRA, using FSDP on a single node; chat or original-source datasets can also be swapped in. The toolkit includes automatic hyperparameter tuning. The authors say very small hyperparameter differences can remove supposedly tamper-resistant safeguards (Qi et al., 2024b). The case setup is in Appendix A.1: batch size 16, LoRA rank and alpha both 64, applied to the q, k, v, o projections. Early stopping patience is 3 on the small dataset, and 1,000 benign examples are trained for one full epoch.
   • Two data types: The harmful set has 51 training and 20 evaluation requests, with compliant answers, covering market manipulation, ecological bioweapons, and assassination weapon design. The benign set comes from PKU-SafeRLHF, keeping only safe answers, with 50 and 1,000 training examples and 20 evaluation examples, non-overlapping with each other.
   • Refusal ablation: Average activations are computed per layer for harmful and harmless prompts, prompts whose next token is most likely a refusal are removed, and the refusal direction is obtained via difference-in-means; the direction r* with the lowest refusal rate that does not change the output distribution on harmless prompts is selected, then orthogonalized into the residual stream weights: W′ ← W − r^∗r^∗⊤W/|r^∗|². This makes the residual stream zero in the r* direction. No training is needed. Full details point to Arditi et al. (2024) Section C.
   • Compute limits: With half precision and roughly up to 300B parameters, fine-tuning and ablation can be done on a single node with 8×H100. The toolkit does not support multi-node or quantized models for these two operations.

   Three-dimensional evaluation

   • Compliance: DeepSeek-R1-Distill-Llama-70B first generates topics and open-ended questions for malicious cyberattacks and bioweapons, and Llama Guard removes questions judged safe. This yields 283 biology, 312 chemistry, and 516 cyber questions. Answers are not released, only questions and Llama Guard scores. After answering, StrongREJECT judges compliance or refusal. The authors call this judge a state-of-the-art LLM harmfulness judge; the paper does not state the underlying model used this time, nor numerical thresholds.
   • Knowledge: WMDP has nearly 4,000 biology, chemistry, and cyber multiple-choice questions, which are largely harmless in themselves. For each question, the probability of the first answer token being A, B, C, or D is computed and the maximum is taken; even if the most likely continuation is a refusal, an option is still extracted.
   • Quality: 100 harmless open-ended questions from Freebase QA, scored by Claude-3.5-Haiku. Users can also swap in other open-ended dangerous question sets or JSON multiple-choice questions.

   Models supported by default

   • Out of the box, it lists Llama-3-Instruct at 1B, 3B, 8B, 70B, 405B, and Qwen-2.5-Instruct at 0.5B, 1.5B, 7B, 32B, 72B. Removing safeguards requires the model to be a subclass of AutoModelForCausalLM; evaluation requires vLLM compatibility. Refusal ablation additionally requires the user to provide a function that returns all modules writing to the residual stream. Inference supports tensor and pipeline parallelism.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On Llama-3 and Qwen-2.5, compliance rates rise after removing safeguards, while WMDP accuracy largely holds; small Qwen already has high original compliance.

   Experimental setup

   • Models tested: Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Llama-3.1-8B-Instruct, Llama-3.1-70B-Instruct, Llama-3.3-70B-Instruct, Llama-3.1-405B-Instruct; Qwen2.5-0.5B-Instruct, Qwen2.5-1.5B-Instruct, Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, Qwen2.5-72B-Instruct.
   • Conditions: Original model, refusal ablation, Harmful (51), Benign (50), Benign (1000). Learning rates and steps are in Table 2 and Table 3.
   • Metrics: WMDP likelihood accuracy (Bio, Chem, Cyber); compliance rates on Bio-Chem-Cyber Propensity; the product of the two is effective dangerous capability; generation quality on Freebase QA 100.
   • Judging: Compliance uses StrongREJECT; the paper does not specify the underlying model or thresholds. Quality uses Claude-3.5-Haiku; the paper does not specify scoring details.
   • Scale and intervals: WMDP has nearly 4,000 questions; propensity sets are 283 biology, 312 chemistry, 516 cyber; Freebase QA subset is 100. Bar plots show means and 95% bootstrap confidence intervals (n=5000). The paper does not report repetition counts for each cell in the main tables.
   • 405B: On 8 H100s, the toolkit cannot directly remove guardrails from Llama-3.1-405B. The case study performs SFT outside the toolkit with FP8 and DeepSpeed ZeRO stage 2, and adds LoRA to pre-quantized weights for evaluation; refusal ablation was not produced. Evaluation itself still uses the toolkit.

   Main results

   The table below shows WMDP-Bio accuracy for original models and Bio Propensity compliance rates. Based on Table 2 and Table 3. The main table has 8 rows, omitting Llama-3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, and Qwen2.5-32B-Instruct.

   Swipe horizontally to read the table

   Model	Original accuracy	Original compliance	Ablation compliance	Harmful (51) compliance	Benign (1000) compliance
   Llama-3.2-1B-Instruct	0.57	0.02	0.67	0.58	0.26
   Llama-3.1-8B-Instruct	0.73	0.00	0.98	0.73	0.86
   Llama-3.1-70B-Instruct	0.83	0.39	1.00	0.81	0.70
   Llama-3.3-70B-Instruct	0.82	0.18	0.99	0.78	0.66
   Llama-3.1-405B-Instruct	0.85	0.03	Not reported	0.80	0.77
   Qwen2.5-0.5B-Instruct	0.51	0.74	Not reported	0.51	0.59
   Qwen2.5-7B-Instruct	0.75	0.43	0.99	0.98	0.91
   Qwen2.5-72B-Instruct	0.84	0.21	0.98	0.92	0.94

   • The authors state that WMDP accuracy increases with scale, and removing guardrails does not change knowledge much; the main text writes that Llama's WMDP-Bio goes from about 55% at 1B to 85% at 405B, with Table 2 original models corresponding to 0.57 and 0.85. Table 3 has an exception: Qwen2.5-0.5B after harmful fine-tuning has WMDP-Bio from 0.51 to 0.45, the lowest Bio accuracy in both tables.
   • The authors write in Section 4.2 that original models typically comply with dangerous requests below 5%, and after removing guardrails up to about 95%, and state that Llama-3.3-70B-Instruct has higher compliance. In Table 2, that original model has Bio compliance of 0.18; Llama-3.1-70B-Instruct original is 0.39. After ablation, many are above 95%, such as 8B's 0.98 and Llama-3.1-70B's 1.00. The authors state that Llama-3.1-70B is omitted from figures because it overlaps with Llama-3.3-70B.
   • In Table 3, Qwen2.5-0.5B and 1.5B original Bio compliance are 0.74 and 0.75, higher than 32B's 0.20 and 72B's 0.21. Harmful fine-tuning on 0.5B reduces Bio compliance from 0.74 to 0.51; benign 50 on 1.5B reduces Bio compliance from 0.75 to 0.62. The authors do not separately explain these two decreases.

   Effective dangerous capability

   • The authors multiply WMDP accuracy by compliance rate. Figure 1 only plots the Llama family with harmful 51-sample fine-tuning: the y-axis Effective Dangerous Capabilities goes up to 0.7, the x-axis Model Size (Parameters) is from 10^9 to 10^11, with series Original, Harmful Finetune (51 samples), Safety Gap. The main text does not give point-by-point values.
   • Figure 3 covers the Llama biology subset for original models, two benign fine-tunings, harmful fine-tuning, and refusal ablation. The authors state that after removing guardrails, this quantity increases with scale, and most original models are below 0.05, with the exception of Llama-3.3-70B-Instruct. The authors state the same pattern appears for Qwen-2.5 (Figure 8). The small Qwen original compliance rates in Table 3 above are not on the same order as the generalization that "most original models have products below 0.05".
   • The authors conclude: evaluating only guardrailed models misleads estimates of risk from malicious users; larger models give more accurate dangerous answers after guardrails are removed; existing guardrails mainly change how knowledge is used, not the knowledge itself.

   Generation quality

   • Figure 4 uses Claude-3.5-Haiku to score Llama responses on harmless Freebase QA questions. The authors state that supervised fine-tuning typically lowers scores by 10 to 30 percentage points relative to original models, with the gap shrinking with scale; refusal ablation is close to original models. In Table 2, Llama-3.1-8B original quality is 0.93, harmful fine-tuning 0.70, ablation 0.92; 405B original 0.97, harmful fine-tuning 0.92, benign 1,000 0.89.
   • In Table 3, Qwen2.5-72B original quality is 0.97, ablation 0.97, harmful fine-tuning 0.89, benign 1,000 0.84. Qwen2.5-0.5B original 0.69, harmful fine-tuning 0.55.
   • The authors believe that looking only at multiple-choice accuracy misses quality degradation, hence the need for multiple evaluations; they also believe refusal ablation may be more targeted and less intrusive to harmless response quality. The authors also see some trade-off between harmful compliance and response quality along the fine-tuning path.

   Chemistry and cyber

   • Table 2: Llama-3.1-405B original WMDP-Chem 0.73, WMDP-Cyber 0.62; original compliance across the three categories is Bio 0.03, Chem 0.06, Cyber 0.05. After harmful fine-tuning, compliance is 0.80, 0.86, 0.87. Benign 50 Cyber compliance is 0.38, lower than its Bio compliance of 0.60.
   • Table 2: Llama-3.1-8B after ablation has Chem and Cyber compliance of 0.99 and 0.98. Llama-3.3-70B after ablation has Cyber compliance of 1.00.
   • Table 3: Qwen2.5-0.5B original Chem compliance is 0.78. Qwen2.5-72B original compliance across the three categories is 0.21, 0.20, 0.21; after ablation it is 0.98, 0.99, 0.99; after benign 1,000, WMDP-Cyber goes from 0.63 to 0.54.

   Other ablations and analyses

   • Benign 1,000 in both Table 2 and Table 3 uses 62 steps; small dataset steps vary with early stopping, e.g., Llama-3.1-8B harmful fine-tuning 8 steps, 405B harmful fine-tuning 20 steps.
   • Qwen2.5-32B after ablation has Bio, Chem, Cyber compliance all 1.00 (Table 3); this model is not in the main table.
   • Qwen2.5-72B after benign 1,000 has WMDP-Bio from 0.84 to 0.80 and WMDP-Cyber from 0.63 to 0.54 (Table 3).
   • Qwen2.5-32B after benign 1,000 has WMDP-Cyber from 0.61 to 0.53 (Table 3).
   • Llama-3.1-70B original Bio, Chem, Cyber compliance is 0.39, 0.41, 0.19, higher than Llama-3.3-70B original in the same table at 0.18, 0.21, 0.12 (Table 2).
   • Appendix A.2: Refusal ablation on Qwen2.5-0.5B-Instruct could not find a direction that simultaneously has a large refusal difference and small KL divergence on harmless instructions; a high-quality ablation of Llama-3.1-405B-Instruct also could not be produced.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors say only two model families are covered, the dangerous capability estimate is a proxy, and the toolkit cannot yet directly handle Llama 405B.

   Limitations and future directions noted by the authors

   • Additional supervised fine-tuning and refusal ablation datasets could be added. The authors give an example: applying jailbreaks to fine-tuning data increases unalignment (Bowen et al., 2024) (6.1).
   • Due to information hazard restrictions, datasets containing instructions for weapons of mass destruction cannot be publicly released; the authors believe dangerous capability evaluation could be extended, including non-public proprietary datasets (6.1).
   • The authors state the toolkit could also include general capability evaluations, such as Gao et al. (2024) (6.1).
   • On the 8 H100s used by the authors, the toolkit cannot directly remove guardrails from Llama 405B. The authors state that native support for quantized models might extend to over one trillion parameters (6.1).
   • Case studies only cover Llama-3.1 and Qwen-2.5 families; the authors state that determining whether the "gap widens with scale" generalizes requires more model families. These models also have limited guardrails, such as supervised fine-tuning and direct preference optimization. The safety gap under tamper-resistant guardrails (TAR, RepNoise) remains unclear; the authors state that these guardrails are relatively fragile in Qi et al. (2024b) (6.2).
   • Effective dangerous capability is only a proxy: dangerous question compliance rate times benign question accuracy. The authors acknowledge that if a model is less capable at answering dangerous questions, this proxy may overestimate (6.2).

   Experimental coverage

   • Tested are 6 Llama-3 Instruct variants and 5 Qwen2.5 Instruct variants; numbers are in Table 2 and Table 3. The main text case study focuses on the Llama biology subset; chemistry, cyber, and full Qwen results are in Appendix A.
   • Methods for removing guardrails are harmful fine-tuning (51), benign fine-tuning (50 and 1,000), and refusal ablation. The paper states that refusal ablation results could not be produced for Llama-3.1-405B-Instruct and Qwen2.5-0.5B-Instruct (Section 4, Appendix A.2).
   • Evaluations are WMDP likelihood accuracy, Bio-Chem-Cyber Propensity compliance rates (StrongREJECT), and Freebase QA 100 quality (Claude-3.5-Haiku). Bar plots use bootstrap n=5000. The paper does not report repetition counts for each cell in the main tables.
   • The paper does not report the underlying model for StrongREJECT in this study, nor agreement numbers between judges and human judgments in this experiment.
   • The toolkit does not support multi-node or quantized model unalignment by default. The 405B SFT is an FP8 and DeepSpeed ZeRO stage 2 implementation outside the toolkit on 8×H100; evaluation of that model still uses the toolkit (Section 4, Appendix A.1.1).

   Collapse this answer

6. 06Summarize the main content of the paper

   The Safety Gap Toolkit estimates the difference in effective dangerous capability before and after removing safeguards using the product of the compliance rate and WMDP accuracy.

   Safety Gap Toolkit is used to quantify the additional effective dangerous capability of open-weight models after safeguards are removed.

   • Problem: Production versions are often trained to refuse harmful requests, but supervised fine-tuning and orthogonalization of the refusal direction can remove these safeguards. The authors want to compare not the default refusal rate itself, but how much more accurate and executable dangerous guidance the model can provide before versus after removing safeguards.
   • Estimation: The authors say a benchmark of high-risk open-ended questions with expert answers does not exist and should not be made public. The toolkit instead uses the compliance rate on a self-built propensity set, multiplied by the likelihood accuracy on WMDP multiple-choice questions. Safeguard removal includes harmful fine-tuning (51 examples), benign fine-tuning (50 or 1,000 examples), and refusal ablation. Freebase QA 100 is also used to assess harmless answer quality.
   • Llama: In Table 2, Llama-3.2-1B original WMDP-Bio is 0.57, and Llama-3.1-405B is 0.85. Llama-3.1-8B original Bio compliance is 0.00, 0.98 after ablation, 0.73 after harmful fine-tuning, and 0.86 after 1,000 benign examples; corresponding WMDP-Bio is 0.73 and 0.74. After harmful fine-tuning, 405B Bio compliance is 0.80, with no ablation figure.
   • Not uniformly so: In Table 2, Llama-3.1-70B original Bio compliance is already 0.39. In Table 3, Qwen2.5-0.5B original Bio compliance is 0.74, and after harmful fine-tuning WMDP-Bio drops from 0.51 to 0.45. The authors still say effective dangerous capability after removing safeguards rises with scale, and the safety gap widens.
   • Quality: The authors say supervised fine-tuning typically lowers Freebase QA quality by 10 to 30 percentage points, while refusal ablation stays close to the original model. In Table 2, 8B original quality is 0.93, 0.70 after harmful fine-tuning, and 0.92 after ablation.
   • Authors' conclusion: Evaluating only the safeguarded version underestimates the risk malicious users can mobilize after open-weight release; existing safeguards mainly lower compliance rather than removing dangerous knowledge. The authors hope the toolkit promotes more tamper-resistant safeguards and welcome community extension of methods and evaluations.

   Collapse this answer

Related papers