Policy is not capability: some safety benchmarks measure the definition of unsafety you adopted during training

Current safety guardrail evaluation treats "unsafe" as an inherent objective property of text, thereby conflating the model's true safety detection capability with the specific moderation policy absorbed by the model.

• Disconnect between evaluation scenarios and deployment reality: The authors point out that papers typically directly report a guardrail's macro F1, capture rate, or over-refusal rate on public benchmarks, leading readers to make horizontal comparisons across models; but in actual deployment, whether "reproducing copyrighted lyrics" or "telling a dirty joke" is a violation depends entirely on the platform's moderation policy rules, and is not an objective fact of the text itself.
• Core assumption flaw of existing evaluation protocols: Existing protocols implicitly assume that evaluating on the same prompts means comparing the same question; the authors believe this assumption does not hold in deployment, and that a considerable portion of what benchmark evaluations report is in essence answering "whose policy the model's training absorbed."
• Research opportunity and core observation: When developing an on-device safety guardrail, the authors had the same batch of approximately 53,000 prompts annotated successively through two pipelines, and the two showed systematic disagreement on 1/6 of the prompts. The disagreement stems from differences in problem definition: policy A asks "would a human moderator block this," while policy B asks "is this a request for harmful content."
• The paper's positioning as a diagnostic study: The authors treat this phenomenon as a measurement tool for diagnosing evaluation protocols, fixing the network architecture and optimizer, constructing 10 branches with different training paths based on the two sets of labels and training at two parameter scales, and systematically evaluating benchmark data contamination, policy sensitivity span, the manipulability of policy by a small number of parameters, and bias in validation set evaluation.

In the introduction and related work, the authors discuss the following three groups of research, and separately give the baseline and benchmark settings:

Open-source safety guardrails and taxonomies

• LlamaGuard (2023), ShieldGemma (2024), WildGuard (2024), and Aegis (2024) — established the paradigm of fine-tuned classifiers as safety guardrails, all with fixed taxonomies; the authors cite independent evaluations pointing out that LlamaGuard's custom category parameter is mostly decorative in practice.
• The Granite Guardian team (Padhi et al., 2025; Nagireddy et al., 2025) — reported that the definitional difference between "using" and "mentioning" harmful content was the main error source for its early social bias detector, and encoded it through taxonomy-driven synthetic data.
• Qwen3Guard (2025) — introduced three-class labels Safe, Unsafe, and Controversial in its output; the authors state that this category is similar to the controversial middle class BOUNDARY_SAFE in this paper's taxonomy, and that its strict/relaxed modes are post-training human mapping decisions.

Annotator disagreement and construct validity

• Pavlick and Kwiatkowski (2019), Plank (2022), Uma et al. (2021), Goyal et al. (2022), and Sap et al. (2022) — established the existence of inherent disagreement in human annotation and the costs of directly averaging disagreement.
• Jacobs and Wallach (2021), Bowman and Dahl (2021), Liao et al. (2021), and Raji et al. (2021) — explored construct validity in machine learning evaluation; the authors point out that in the safety guardrail field, the assumption that "unsafe is independent of annotators" was never stated and does not hold.

Benchmark contamination and representation transfer mechanisms

• Young (2025) — evaluated the robustness of 10 released guardrails against 21 categories of attacks, finding that detection rates all dropped against novel attacks (Qwen3Guard-8B from 91.0% to 33.8%); the authors point out that this evaluation is a behavioral-level inference, whereas this paper directly measures corpus text overlap.
• Knowledge distillation and parameter fine-tuning research (Hinton et al., 2015; Hooker et al., 2020; Ojha et al., 2023; Kirichenko et al., 2023) — demonstrated that knowledge distillation transfers bias and calibration, and that retraining the last layer can adjust a model's behavior on spurious correlations.

Comparison baselines and evaluation benchmarks

• Baseline methods: Qwen3Guard-Gen (0.6B, 4B, 8B) and Granite Guardian 3.1-2B.
• Benchmark datasets: 15 public benchmarks (33,137 prompts in total), including HarmBench (standard, contextual, copyright), AdvBench, JailbreakBench (harmful, benign), StrongREJECT, DoNotAnswer, WildGuard (harmful, benign), SaladBench (base, attack, defense), OR-Bench (toxic, hard).

The authors state that this paper's contribution is diagnostic rather than methodological; it does not propose a new guardrail or benchmark, but by isolating two annotation policies on the same batch of data, systematically demonstrates how policy choice dominates benchmark scores and validation set ranking when adjusting only 0.03% of parameters or through distillation soft labels, and measures corpus contamination and seed noise floor ignored by existing evaluations.

The authors systematically analyze the mechanism by which policy definitions enter the model and their impact on benchmark evaluation, by constructing a multi-branch controlled experimental framework and using dual-annotated data for the same batch of prompts in an industrial setting.

Task system and dual-annotation policy construction

The guardrail model receives a single user prompt and predicts 4 coarse categories (GENERIC_SAFE, BOUNDARY_SAFE, HARMFUL, COMPOUND), with binary unsafe defined as the union of HARMFUL and COMPOUND; it simultaneously predicts a multi-label vector covering 24 pillars.

• Policy A: Constructed based on human moderation guidelines, judging copyright reproduction, policy refusal requests, and borderline content as blockable.
• Policy B: Based on a review committee composed of 4 large instruction models, through three rounds of binary cascaded judgment, judging as unsafe only requests with genuine harmful intent.
• The two policies annotate the same prompt pool (training set 53,441; validation set 5,936; multilingual validation set 7,662), with a complete agreement rate of 83.6% for coarse categories and 90.1% for binary unsafe; disagreements are mainly concentrated in the BOUNDARY_SAFE and COMPOUND categories.

Model architecture and three policy injection paths

The model uses the ModernBERT encoder architecture, including a ModernBERT-large teacher model with 394.8M parameters and a ModernBERT-base student model with 149.0M parameters, with three linear classification heads (4-class head, 24-pillar head, harmful text span head) attached after the [CLS] token state. Three paths by which policy definitions enter the model (Figure 1):

1. Training pool hard label input: directly fine-tuning the entire encoder through data;
2. Distillation teacher soft label transfer: the student model absorbs the teacher's prediction distribution during training through a knowledge distillation loss with temperature coefficient τ=4;
3. Classification head independent calibration: freezing the backbone encoder and retraining only the three linear classification heads on a small calibration set (0.04M parameters, 0.03% of the student model's 149M parameters).

Design of 10 experimental branches

The authors designed 10 experimental branches (E1 to E10), sharing the network architecture, codebase, and optimizer family:

• Data mixing branches (E2–E5): mixing policy A data in different proportions in the training pool, where E3 and E5 additionally introduce explicit policy refusal anchor samples;
• All-policy-A hard label branch (E6): all training set hard labels are policy A;
• Reverse mixing branch (E9);
• Pure teacher distillation branch (E8): all student model hard labels are policy B, but distillation uses a teacher model fine-tuned with policy A;
• Classification head calibration branches (E1, E7, E10): freezing the backbone network and retraining classification head parameters only on 8,489 (E1, E10) or 53,395 (E7) samples.

Formalization of sensitivity metrics and statistical tests

To quantify the degree to which benchmarks are determined by label policy, the authors define a label supervision span metric: Δ_b = 1/(|M|) ∑_{m ∈ M} [max_a s_b(a, m) − min_a s_b(a, m)] This formula represents the average of the score range of benchmark b across 10 branches at each model scale.

To test the strength of the association between supervision paths and extreme rankings, the authors use a permutation rank test to compute the probability: \Pr[A \text{ occupies the } k \text{ extreme ranks of } 10] = \binom{10}{k}^{-1} This formula represents the exact probability that, under the null hypothesis that branches are exchangeable, a specific supervision path branch set A of size k occupies exactly all k extreme ranks.

Experimental setup

• Tested models: ModernBERT-base (149.0M student model), ModernBERT-large (394.8M teacher model); comparison baselines include Qwen3Guard-Gen (0.6B, 4B, 8B) and Granite Guardian 3.1-2B.
• Benchmark datasets and scale: 15 public benchmarks with a total of 33,137 prompts, covering HarmBench (standard 200, contextual 100, copyright 100), AdvBench (1,094), JailbreakBench (harmful 100, benign 100), StrongREJECT (313), DoNotAnswer (939), WildGuard (harmful 754, benign 945), SaladBench (base 21,318, attack 5,000, defense 200), OR-Bench (toxic 655, hard 1,319).
• Experimental control branches: 10 branches in total, including data mixing (E2–E5), all-policy-A hard labels (E6), pure teacher distillation (E8), classification head fine-tuning (E1/E7/E10), and reverse mixing (E9).
• Evaluation metrics: harm capture rate (CAP, the proportion of harmful samples judged unsafe), over-refusal rate (OVER-REF/OR-H, the proportion of benign samples judged unsafe), multilingual validation set four-class accuracy (VAL), copyright capture rate (COPY), label supervision span Δb, and random seed variance noise floor Φb.
• Decision protocol: read binary unsafe labels (HARMFUL ∪ COMPOUND); in the open-source baseline comparison, Qwen3Guard provides a relaxed mode (Controversial recorded as safe) and a strict mode (recorded as unsafe); this paper's default binary mapping records BOUNDARY_SAFE as safe.
• Repetition and computation settings: each branch uses random seed 42 trained once; the random seed noise floor is evaluated on 37 triples with three random seeds (42/43/44) (111 runs in total); prompt resampling is performed B=2,000 times.

Main results

The table below gives the main results of the 149M student model on the decontaminated subset (according to Table 1; branches E3 and E5 are omitted, both with COPY 0–1 and OR-H 40.3%–56.4%):

Swipe horizontally to read the table

• Terminal supervision signal dominates policy performance: In the 149M student model, the 4 branches with policy A as the terminal signal (E8, E1, E7, E10) all reached a HarmBench-copyright score of 100, with OR-Bench-hard over-refusal rates dropping to 3.4%–6.8%; even when the training data hard labels fully adopt policy A (E6), its score reached only 17, because its output was overridden by the soft labels of the policy B teacher model.
• Very few parameters suffice to switch policy: The E1, E7, and E10 branches, which freeze the backbone encoder and retrain only the classification head (0.04M parameters, 0.03%), achieve policy adoption effects comparable to full model fine-tuning.
• Decoupling of capability and policy performance: The average capture rate (CAP) of each branch on the 6 capability-load benchmarks fluctuates between 80.0% and 91.1%, with no case of losing basic safety detection capability due to policy switching.

Audit of overlap between training corpus and benchmarks

• What was measured: measured the exact-match overlap between a training/validation/calibration pool consisting of 269,861 unique samples and 15 public benchmarks.
• Results: JailbreakBench-benign overlap rate reached 88.0% (88/100), OR-Bench-hard reached 82.1% (1,083/1,319), HarmBench-copyright was 25.0%, WildGuard-harmful was 14.3%, StrongREJECT was 12.1%, while OR-Bench-toxic was 0.0% and AdvBench was 2.0% (Figure 2(a)).
• Authors' interpretation: The authors point out that the highest overlap is with over-refusal benchmarks, stemming from directly scraping public benchmarks when constructing boundary safety samples; after decontamination, the divergence span between benchmarks further widened (HarmBench-copyright from 0.98 to 0.99, OR-Bench-hard from 0.42 to 0.57).

Random seed noise floor and benchmark sensitivity span

• What was measured: measured the pure seed variance noise floor Φb across 37 triples with three random seeds (42/43/44), 111 runs in total, and compared it with the cross-branch label supervision span Δb.
• Results: The Δb of HarmBench-copyright reached 0.993, while the widest seed noise floor median was 58.5 (median of all 37 groups was 4.0); the Δb of WildGuard-benign was only 4.3, and the widest seed noise floor was 5.0 (Table 4). On HarmBench-copyright, the branches that did not adopt policy A scored at most 73, while branches that adopted policy A scored from 74 to 100.
• Authors' interpretation: The authors believe that benchmark sensitivity is entangled with seed instability, and that without measuring the seed noise floor, the middle portion of the benchmark ranking is not interpretable; only a few benchmarks at the two extremes (such as copyright and violent requests) can clearly distinguish policy from noise.

Cross-institution open-source guardrail evaluation and binary mapping effects

• What was measured: compared Qwen3Guard (0.6B, 4B, 8B) and Granite Guardian 3.1-2B on the same decontaminated prompts, and tested the effect of judging the controversial category (Controversial or BOUNDARY_SAFE) as unsafe (Table 2).
• Results: On HarmBench-copyright, Granite Guardian scored 0 and Qwen3Guard-4B scored 99; on OR-Bench-hard, in relaxed mode Qwen3Guard's over-refusal rate was 29.7%–30.5%, Granite's was 65.2%, and this paper's 149M student (E8) was 5.7%. When the controversial category is changed to be judged unsafe, Qwen3Guard-8B's over-refusal rate rose from 29.7% to 48.5%, and this paper's model's over-refusal rate rose from 5.7% to 96.0% (an increase of 90.3 percentage points).
• Authors' interpretation: The authors point out that the 99-point gap between the two open-source guardrails is not a capability difference, but whether the taxonomy defines copyright violations as harm; the mapping rule for the controversial category in post-processing has a greater impact on the over-refusal rate than model retraining.

Other ablations and analyses

• Spearman correlation coefficient between policy A multilingual validation set accuracy and policy-load benchmark scores ρ=+0.74, and with capability-load benchmarks ρ=+0.34 (Section 7).
• 69.3% of the multilingual validation set samples exactly overlap with the policy A training file, and the monolingual validation set overlap rate is 0.2% (Section 7).
• The policy B teacher training samples (138,805) are 2.06 times those of the policy A teacher (67,352), among which 77.1% of samples do not appear in the policy A teacher pool (Appendix A).
• When the calibration set contains only disagreement samples, the four-class accuracy drops to 57.9%, and after adding 3,000 GENERIC_SAFE anchors it recovers to 87.9% (Appendix A).
• The LLM review committee has Fleiss κ=0.78 for binary safety rounds, and Fleiss κ=0.70 for boundary safety and generic safety sub-rounds (Appendix A).

Negative results and exceptions

• At the 149M scale, full fine-tuning with all policy A hard labels (E6) reached only 17 on HarmBench-copyright, failing to successfully install policy A;
• At the 395M scale, the rank test grouping failed when directly applying the 149M definition (hitting only 3/4 and 2/4), and the authors reached statistical level only after modifying the grouping definition (Section 6);
• The E1 branch terminated early after training 2 epochs, but the authors point out that this undertrained branch is still located at the extreme end (Appendix A).

Limitations pointed out by the authors and future directions

• Single random seed and noise floor proxy limitation: Each experimental branch was trained with only a single random seed; the seed noise floor comes from neighboring training recipes rather than the branches themselves and cannot be directly tested, causing the middle portion of the benchmark sensitivity ranking to be unexplainable with certainty (Section 9).
• Deficiencies in the training setup for classification head calibration: The head classification fine-tuning branches did not set random seeds, and checkpoints were selected based on training set loss rather than validation set loss (Section 9, Appendix A).
• Coverage of model architecture and taxonomy: The evaluation is limited to 20 ModernBERT encoder models, one proprietary 24-pillar taxonomy, and two policy samples, and the two policies differ simultaneously in annotation modality (human vs LLM committee) and decision criteria, so the specific ranking may not transfer directly (Section 9).
• Sample size and collinearity of training paths: The training pool sizes of the branches are inconsistent (spanning from 8,489 to 144,744), and in the 395M model there is collinearity between data pool size and training path (Section 9).
• Copyright benchmark templates are too homogeneous: The 75 prompts of HarmBench-copyright after decontamination consist of only two prompt templates (39 lyric generation and 36 long passage generation); the 0.99 span reflects the performance of a single controversial rule under a small sample, rather than a broad construct (Section 9).
• Asymmetry of baseline comparison: The branches in this paper were developed for these benchmarks, whereas the open-source baseline models were not; data decontamination is defined only based on this paper's data pool, and the baseline models' own training overlap is unknown (Section 9).
• Unresolved out-of-distribution generalization: The actual model precision derived at the deployment threshold is 0.968 and recall is 0.496; the authors explicitly state that this paper does not solve the generalization problem (Section 9).

Experimental coverage

• The tested model systems cover 6 models in total: ModernBERT-base (149M), ModernBERT-large (395M), Qwen3Guard-Gen (0.6B, 4B, 8B), and Granite Guardian 3.1-2B (Section 2, Section 6).
• The evaluation benchmarks cover 15 public safety benchmarks with a total of 33,137 prompts, covering both harm detection and over-refusal scenarios (Section 2, Appendix A).
• The random seed noise floor analysis covers 37 training triples across 9 lineages and up to 6 data versions (111 runs in total) (Appendix C).
• The annotation committee consists of 4 instruction-tuned models with 26B–35B parameters (two Gemma series, two Qwen series), with 3 samples per item for a total of 12 votes (Section 2, Appendix A).
• The paper does not report overlap audit data against each open-source baseline model's own training corpus, and does not report independent repeated training of each branch under multiple random seeds (Section 9, Appendix A).

1. Research positioning and core question: This paper is a diagnostic study of safety guardrail evaluation protocols, aiming to point out that public benchmark scores are often driven by the moderation policy definitions absorbed by the training data, rather than purely measuring the model's safety detection capability.
2. Method and experimental framework: Based on dual-annotated data from the same batch of approximately 53,000 prompts in industrial deployment under human moderation policy A and LLM-judge policy B, the study constructed 10 training branches covering data mixing, pure hard labels, teacher knowledge distillation, and fine-tuning only the classification head (0.03% of parameters), and conducted evaluations at two scales, 149M and 395M, combined with 15 public benchmarks.
3. Main finding one: benchmarks have high overlap on over-refusal sets: In the exact-match audit between the authors' training pool and the benchmarks, the over-refusal benchmark JailbreakBench-benign had an overlap rate of 88.0% (88/100), and OR-Bench-hard reached 82.1% (1,083/1,319). The authors point out that this is due to the training set collecting public benign borderline data.
4. Main finding two: policy definitions and very few parameters dominate benchmark performance: On the decontaminated subset, the HarmBench-copyright score span reached 0.99, while WildGuard-benign was only 0.043; in the 149M student model, only updating the classification head calibration with 0.03% of parameters or introducing policy A teacher soft labels could make the HarmBench-copyright score reach 100 and the OR-Bench-hard over-refusal rate drop to 3.4%–6.8%, with an exact permutation rank test p=0.005.
5. Main finding three: validation set metrics are misleading and post-processing mapping effects: The correlation between the multilingual validation set accuracy annotated by a single set of policy A and policy-load benchmark scores (ρ=+0.74) is much higher than that for capability-load benchmarks (ρ=+0.34); directly mapping the controversial category to unsafe can raise the over-refusal rate from 5.7% to 96.0% (an increase of 90.3 percentage points), an effect exceeding model retraining.
6. The authors' conclusions and recommendations: The authors believe that safety benchmarks themselves do not need to be entirely rejected, but call for future guardrail evaluations to publicly disclose prior data overlap, random seed noise floors, and the specific operational definitions being evaluated as core variables.