Skip to content

Operating points of frozen guardrail probes fail under base model updates

Operating-Point Failure of Frozen Guardrail Probes Under Backbone Updates

arXiv: 其他 AI 安全论文· arXiv:2611.19216· Yusuf Khalid Shire, Sang-Chul KimSchool of Computer Science, Kookmin UniversityIndexed by this site · 2 days ago· Originally publishedDefenses· Its place in the scan record →

Quick read

Analysis and theory

Based on the paper PDF (AI-generated); refer to the original

The authors measure the operating point of frozen guardrail probes under 16 public model updates: at the 1% target, 8 of 9 Phase 1 units fail closed, and the AUROC check misses all of them.

Questions
The guardrail probe's threshold is set at a 1% false positive rate and then frozen; the backbone model is subsequently replaced by a version update not planned by the deployer; AUROC-type staleness checks do not read the threshold and cannot answer whether the operating point (FPR/TPR) survives.
Methods
On 18 headline units from 16 public checkpoint pairs, with 5 probe seeds and 1000 resplits, measure the paired ratio before and after the frozen threshold; propose a two-moment model to explain the change and a label-free estimated-rate check to detect damage, and evaluate repairs such as threshold resetting and realignment as well as poisoning and update attacks.
Experiments and results
At the 1% target, 8 of 9 Phase 1 units fail closed, and in Phase 2, 2 close and 1 opens; the maximum FPR rises from 1.01% to 15.12%, while the AUROC change does not exceed 0.0058 and the Duan standard does not judge any unit stale; the estimated-rate check flags 0.918 of damaged units, the AUC rule only 0.019; resetting the threshold on stored benign prompts restores the target FPR.
Limitations and future work
The authors note that Phase 1 pairs have no registered selection rule, that stage is confounded with update type and readout, that the 7 pairs' configuration changes are not separated from weights, that HH-harmless training data overlaps with evaluation items, that only open-weight models below 9B are studied, and that adaptive attackers are out of scope; the attack and LoRA experiments each use one backbone.
Experimental setupMistral-7B v0.1、Mistral-7B v0.2 et al. · Aegis 2.0、WildGuardMix et al. · paired ratio r(FPRpost/FPRpre)、FPR/target et al.
Threat model
The attacker knows the frozen probe, layer and threshold, and occupies one of three positions: update producer controlling weights, live traffic contributor controlling the share of prompts used to fit quantile transfer, or stored reference set supplier or a write-capable insider controlling the share of prompts used for threshold resetting. The attack moves the frozen scores of all or a single harmful category within the stealth bound; adaptive attackers and attackers who write new prompt text are out of scope.
Evaluated models
Mistral-7B v0.1Mistral-7B v0.2Gemma-2-9BLlama-3-8BLlama-3.1-8BQwen2.5-7BOLMo-2-1124-7BMistral-7B-Instruct v0.1Mistral-7B-Instruct v0.2Mistral-7B-Instruct v0.3Llama-3-8B-InstructLlama-3.1-8B-InstructQwen2-7B-InstructQwen2.5-7B-InstructPhi-3-mini-128k-instructPhi-3.5-mini-instructLlama-3.1-Tulu-3-8B
Benchmarks
Aegis 2.0WildGuardMixXSTestAlpacaHH-harmlessPKU-SafeRLHFARC-Easy
Metrics
paired ratio r(FPRpost/FPRpre)FPR/targetTPR at frozen thresholdAUROCestimated-rate check 的 sensitivity/specificity
AI summaryThe study finds that frozen guardrail probes and thresholds have their operating points fail after base model updates, with the largest increase being a false positive rate rising from 1.01% to 15.12%, while per-unit AUROC changed by no more than 0.0058. The experiments cover 18 units across 16 public LLM updates; at a 1% false positive rate target, 8 of 9 units in the first phase failed as too strict (fail closed), and in the preregistered second phase (descriptive), 3 of 9 failed as too strict or too loose. A benchmark-staleness criterion did not flag any update as stale; an unlabeled ratio estimate flagged 0.918 of ratio-doubling or halving units and passed 0.898 of the remaining units; a two-moment model matched changes within 1.5× on 0.905 of 63 preregistered units and 0.773 of 22 shifted units.

The study finds that frozen guardrail probes and thresholds have their operating points fail after base model updates, with the largest increase being a false positive rate rising from 1.01% to 15.12%, while per-unit AUROC changed by no more than 0.0058. The experiments cover 18 units across 16 public LLM updates; at a 1% false positive rate target, 8 of 9 units in the first phase failed as too strict (fail closed), and in the preregistered second phase (descriptive), 3 of 9 failed as too strict or too loose. A benchmark-staleness criterion did not flag any update as stale; an unlabeled ratio estimate flagged 0.918 of ratio-doubling or halving units and passed 0.898 of the remaining units; a two-moment model matched changes within 1.5× on 0.905 of 63 preregistered units and 0.773 of 22 shifted units.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Whether the operating point of a frozen probe threshold survives a backbone update cannot be answered by an AUROC check; the authors measure it and provide label-free detection and repair.

    Core question: whether the operating point (FPR/TPR) of a frozen guardrail probe threshold still holds after the backbone model is updated, rather than whether AUROC survives.

    • Scenario and importance: The deployer sets the linear probe's threshold T to flag 1% of benign prompts and then freezes it; the backbone is subsequently replaced by an unplanned release; if benign scores cross T, the guardrail fails closed (more false alarms) or fails open (more permissive), while the deployer still believes it flags 1%.
    • Shortcomings of existing methods: The authors state that Duan [1] judges probe staleness by an AUROC standard and records recall at 1% and 5% FPR, but does not report the FPR actually realized at the frozen threshold; AUROC does not read the threshold, and when all scores shift by the same amount, AUROC is unchanged while the operating point can move (Varshney [2] demonstrated this on constructed scores).
    • Core observation: The authors hold that the update does not change the separability of benign and harmful prompts, but shifts the mean and standard deviation of benign scores, thereby moving the false positive rate at the threshold; this change can be characterized by the first two moments.
    • Paper outputs: Operating-point measurements on 18 headline units, a two-moment explanatory model, a label-free estimated-rate check, label-free repairs such as threshold resetting, poisoning and update-control attacks on the repair reference set, and a record of 137 registered predictions.
    • Threat model (Section 3):
      • Goal: Move the operating point of the frozen probe—the scores of all prompts, or the scores of a single harmful category.
      • Knowledge: The attacker knows the frozen probe, its layer, and the threshold T.
      • Capability: One of three positions—an update producer controlling the weights; a live traffic contributor controlling the share of prompts used to fit quantile transfer, etc.; a stored reference-set supplier or a writable insider controlling the share of prompts used for threshold resetting. The attack is within the stealth bound (KL no more than the KL between clean seeds of 0.08644 nats/token, ARC-Easy accuracy within the clean range).
      • Victim system: A single frozen linear probe guardrail and its threshold; adaptive attackers and attackers who write new prompt text (e.g., trigger phrases) are out of scope.
  2. What related research exists?

    Related work falls into four groups: probe guardrails, probes under model change, attacks, and recalibration under distribution shift; the authors position the difference as measuring the operating point at a frozen threshold.

    Probes as guardrails

    • Belinkov [7] reviewed hidden-state probing methods; Sahoo et al. [8] reported that embedding drift reduced a toxicity classifier's AUROC to chance while confidence remained high.
    • Goldowsky-Dill et al. [9] reported that a 1% FPR threshold missed 4% of deceptive responses on one dataset; McKenzie et al. [10] compared high-risk interaction probes with a language model baseline at that rate.
    • Gao [11] measured detection and false-positive rates achieved by a locked threshold on a frozen model's own calibration/test split; the authors note that this paper reads the same prompts before and after updates.

    Probes under model change

    • Duan [1] used the AUROC criterion to judge whether activation monitors were stale under quantization, LoRA, merged LoRA and QLoRA, and repaired them with realignment using paired unlabeled activations; the authors note that it does not give the FPR achieved at a frozen threshold, and that this paper's estimated-rate check estimates the post-update FPR ratio rather than the AUROC change.
    • Parallel work [5] held a 5% FPR threshold on post-training checkpoints of OLMo-3-7B and Apertus-8B and reported its drift, resetting the threshold on benign inputs; the authors state that this paper holds a 1% target, covers version increments and measures the poisoning cost. Tsatsos [6] used AUC to judge that base probes transfer to instruction-tuned and distilled successors.
    • Bakker and Louizos [17] froze probes across Qwen2.5 quantization/LoRA/QLoRA; Das and Fioretto [19] recommend retraining probes after fine-tuning and recalibrating after updates; parallel work [18] found that AUROC retained under compression does not guarantee retained refusal.

    Attacks

    • Lyons Keenan et al. [21] used rank-one LoRA to make the model edit activations along the monitored direction to lower the monitor's TPR; the authors state that this paper's A1m and A2 borrow this mechanism, delivered by the update producer, and that A2 moves only one harmful category under a KL constraint.
    • On poisoning, Rubinstein et al. [24] and Kloft and Laskov [25] target anomaly detectors, Wu et al. [26] targets test-time adaptation, and Huang et al. [29] targets the update loop of jailbreak detection classifiers; this paper poisons the repair's reference set.

    Recalibration under distribution shift

    • Berlin [30] refits calibration of the entire benign FPR curve at each release; the authors state that this paper only resets a single threshold on stored benign prompts. Schirmer et al. [31] and Tong et al. [32] give false-positive bounds; the authors note that pre-update scores do not satisfy their assumptions.
    • Lipton et al. [33] and Saerens et al. [34] address label shift; AdaBN [36] re-estimates batch normalization statistics on the new distribution, and this paper's restandardisation applies the same principle to the standardization statistics of probe inputs.

    Baselines and benchmarks: detection baselines are the AUC rule formed by Duan's three criteria, the KS test [51], ATC [52] and the direct ratio; repair controls include quantile transfer, restandardisation, and ridge/Procrustes realignment; datasets are Aegis 2.0 [42], WildGuardMix [43] and XSTest [47].

    Author positioning: this paper reads the FPR at a frozen threshold rather than AUROC, and registers predictions, measures the poisoning cost and controls for updates in attacks.

  3. How does the paper address this problem?

    Freeze the probe and threshold, and measure the paired ratio across resplits before and after the update; explain with the first two moments, detect with a label-free check, repair with label-free restoration, and run poisoning and update attacks.

    Overall approach: freeze the probe, standardization statistics, layer L* and threshold T entirely, measure operating-point changes before and after 16 sets of public checkpoint updates, then explain with a two-moment model, detect label-free with an estimated-rate check, and repair with label-free methods such as threshold resetting; finally attack the updates and repairs in a registered-prediction manner.

    Probe, layer and threshold (Section 4.1)

    • Probe: ℓ2-regularized logistic regression, with input being the layer ℓ hidden state averaged over tokens and standardized; each probe is trained on 4000 prompts (2000 harmful, 2000 benign), with 20% held out for validation for each of 5 seeds.
    • Selection: C and L* are selected jointly by average held-out AUROC of the pre-update model, with C chosen from 8 candidate values and L* every other layer; the selected C are all 10⁻³ or 3·10⁻³.
    • Threshold: T is the smallest distinct score such that at most 1% of calibration negatives score no lower than t; Phase 1 uses 1000 calibration negatives, Phase 2 uses 5000; the 5% threshold is a registered sensitivity arm.
    • Estimator: 5 probe seeds per cell, 1000 re-splits of calibration and evaluation negatives, resetting T each time and then averaging; bands are the 2.5/97.5 percentiles.

    Update pairs (Section 4.2)

    • Phase 1: 7 headline pairs (6 base→instruct releases plus Llama-3-8B→Llama-3.1-8B), of which the Mistral and Llama-3 pairs are repeated on WildGuardMix, for 9 L* cells in total; probes use Aegis 2.0, evaluation split 2000 negatives + 2000 positives; input is the raw prompt without a chat template.
    • Phase 2: 9 instruct-side updates (4 releases, 5 post-training steps), fixed by a registered rule before any Phase 2 scoring; only WildGuardMix is used, with 5000 calibration negatives and evaluation of 6000 negatives + 4000 positives; instruct models use their respective chat templates, averaging only over user-message tokens.
    • Phase 3: model-side attacks and defenses on Mistral-7B-Instruct-v0.2; the clean update is 400 steps of rank-16 LoRA on Alpaca, with 3 seeds. Separately, Mistral-7B v0.1 is LoRA fine-tuned on Alpaca, HH-harmless and PKU-SafeRLHF for 25/100/400 steps.

    Metrics and damage rules (Section 4.3)

    • paired ratio r: the ratio of post-update FPR to pre-update FPR on the same evaluation negatives (FPR floored at 1/n_neg), averaged over seeds and re-splits; r ≥ 2 is failure closed, r ≤ 0.5 is failure open.
    • AUC rule: Duan's three criteria—pre-update AUROC ≥ 0.75 as the qualification gate, ∆AUROC ≤ −0.05 as a big drop, and post-update AUROC < 0.70 as operational failure; this paper applies them to the mean over 5 seeds.
    • Registration: each result is labeled registered (with a verdict) or descriptive; all 137 registered entries are reported (Supplement S-2).

    two-moment model and estimated-rate check (Section 4.4)

    • two-moment model: maps the pre-update score x to f̃(x) = μ1 + (fpre(x) − μ0)σ1/σ0, i.e., a linear mapping using the mean and standard deviation of evaluation negatives before and after the update, testing whether the first two moments can explain the FPR change; error e = |log R − log R̂|, with e ≤ log 1.5 meaning within ×1.5. It uses the post-update moments of evaluation negatives and is explanatory rather than predictive.
    • estimated-rate check: uses only the pre- and post-update moments of stored calibration negatives, comparing the ratio of Gaussian tails at the threshold Gd (Equation 3), alarming when G_d ≥ 2 or ≤ 0.5; calibration and evaluation negatives are disjoint in every split, so it is an out-of-sample check.

    Repairs, attacks and canary (Section 4.5–4.6)

    • Repairs (none require new labels): threshold resetting (Rsame uses 1000 stored calibration negatives, Rfresh uses held-out fresh negatives), quantile transfer (1000 unlabelled traffic items with 1% harmful), restandardisation (class-balanced or traffic reference), ridge/Procrustes realignment.
    • Attacks: A1 adds to each post-update state a translation along the probe direction δ = cσv/(vᵀv), leaving AUROC unchanged; A1m implements this translation with rank-1 LoRA; A2 moves only one target category (cat-1 to cat-3, c ∈ {−2, −1}, 5 seeds each, 30 attacked models in total), with a KL term keeping the model close to the clean update; A3 poisons the repair's 1000-item reference set at a share ρ (0.001 to 0.25).
    • canary: 47 to 99 held-out labeled harmful prompts per target category, with the statistic being the largest drop in canary TPR at the frozen threshold across the three categories, and threshold τ matched to the clean false-positive count of the global estimated-rate check; the canary is fixed only after A2 scoring and is a post hoc check.
  4. What experiments did the paper conduct?

    Measured on 18 headline units: in Phase 1, 8/9 fail closed while AUROC does not move; the estimated-rate check detects 0.918, and threshold resetting restores the target FPR.

    Experimental setup

    • Subjects: 16 public checkpoint pairs (seven Phase 1 pairs, nine Phase 2 pairs), involving 7B–9B open-weight models such as Mistral-7B, Gemma-2-9B, Llama-3/3.1-8B, Qwen2/2.5-7B, OLMo-2-1124-7B, Phi-3-mini/3.5-mini, and Llama-3.1-Tulu-3-8B; additionally, 9 LoRA fine-tunes of Mistral-7B v0.1 and attack experiments on Mistral-7B-Instruct-v0.2.
    • Data: Probe training uses Aegis 2.0 (Phase 1) and WildGuardMix (Phase 2 and two repeated Phase 1 units); XSTest's 250 safe prompts measure over-flagging; LoRA corpora are Alpaca, HH-harmless, and PKU-SafeRLHF (6400 examples each); ARC-Easy's 2376 questions serve as a capability check.
    • Scale and repetition: 5 probe seeds and 1000 calibration/evaluation negative resplits per unit; 200 runs for the reference pool, attacks, and ridge realignment, 50 for Procrustes; 5000 bootstrap iterations.
    • Metrics: paired ratio r (damage cutoffs 2 and 0.5), FPR/target, TPR at the frozen threshold, AUROC; detectors score the damaged class by sensitivity/specificity.
    • Verdict method: No LLM review; AUROC, CKA, etc. are read on a fixed evaluation split; results are labeled registered (HELD/FAILED/REFUTED, etc.) or descriptive.
    • Baselines: AUC rule (Duan's three criteria), KS test, ATC, direct ratio; repair comparisons include quantile transfer, restandardisation, and ridge/Procrustes realignment.

    Main results

    According to Table 3 (1% target, r is the mean over 1000 resplits):

    Swipe horizontally to read the table

    IDDatarCategoryTPR pre→postAUROC pre→post
    1aAegis17.377closed0.396→0.7910.9170→0.9112
    1aWGM9.521closed0.713→0.8940.9704→0.9665
    1bAegis6.783closed0.413→0.6810.9199→0.9213
    1eAegis1.577within0.371→0.4270.9093→0.9133
    1gAegis3.224closed0.420→0.5700.9163→0.9161
    2aWGM0.055open0.738→0.3650.9776→0.9733
    2bWGM2.254closed0.773→0.8390.9814→0.9808
    2eWGM5.874closed0.710→0.8980.9730→0.9760

    (Omitting the two 1c units, 1d, 1f, 2c, 2d, and 2f–2i, all within or closed with smaller magnitude; the full 18 rows are in Table 3.)

    • The authors report: at the 1% target, 8 of 9 Phase 1 units fail closed, and in Phase 2, 2 close (2b, 2e) and 1 opens (2a); the largest increase is 1a Aegis, with FPR going from 1.01% to 15.12%, whose AUROC drops only 0.0058, and across the 18 units the AUROC change does not exceed that value, and the Duan standard does not judge any unit stale (descriptive).
    • The authors note that the two phases differ simultaneously in update type and input format, so the difference in counts between phases cannot be attributed to update type; all 10 fail-closed units show increased TPR, and the failure is an increase in false-alarm burden rather than a loss of detection.
    • At the 5% target, 5 units fail closed, 2a opens, and 12 are within; 1c (both datasets), 1d, 1g, and 2b change from closed to within, which the authors describe as units crossing the cutoff of 2 rather than a large effect disappearing (descriptive).

    Mechanism and estimated-rate check (Section 5.3)

    • Two-moment model: Of the 63 registered Phase 2 units, 0.905 are within ×1.5, with a median error of 0.058 (P2-M.2, HELD), a slope of 0.911 [0.806, 1.292] (P2-M.1, HELD), better than the shift-only control's 0.730 (P2-M.3, HELD); among the 22 moved units, it holds for 17 (0.773).
    • Estimated-rate check: On the 156 Phase 1 layer units, it flags 0.918 of damaged units and passes 0.898 of undamaged units (Table 5, Fig. 1; Fig. 1 is a scatter of realised r against G_d, with 0.5 and 2 tangent lines on both axes; on the fixed split, damaged-and-flagged/damaged-not-flagged/undamaged-flagged/undamaged-not-flagged are 97/13/4/42); the AUC rule flags only 0.019 of damaged units (specificity 1.000).
    • Stronger baselines: KS has sensitivity 0.966 but specificity 0.543 (P-C5.1, HELD); ATC has higher specificity but sensitivity 0.479, and its AUROC of 0.965 is higher than G_d's 0.952, causing P-C5.2, which predicted ATC would rank worse, to FAIL; no baseline is no lower than G_d on both (P-C5.3, HELD).

    Repair and poisoning (Section 5.5)

    • Threshold resetting: On Rsame, the mean FPR/target for the 9 Phase 1 units is 0.98–1.02, and on Rfresh 0.96–1.02 (descriptive); TPR is 0.351–0.721 for the updated model at that rate, and recovery of the pre-update TPR is not guaranteed.
    • Restandardisation: Effective only on class-matched references (FPR/target 1.01–1.71 on class-balanced training rows); on 1% traffic it overshoots to 4.74–9.82 (Aegis) and 21.87, 24.36 (WGM), with 8/9 units worse than no repair (descriptive).
    • Poisoning (n=1000 reference set): High-scoring harmful prompts first cause resetting to lose 10 points of TPR at 0.5% (4 pairs) or 1% (5 pairs); at 1%, resetting flags only 0.006–0.017 of harmful prompts; quantile transfer loses 10 points at 1% but still flags 0.153–0.489 (descriptive).
    • Realignment: 27/27 units preserve the extended Duan standard under six arms (P-RA.1, HELD), but the ridge map leaves FPR/target outside ×1.5 in 9/9 units, and Procrustes in 6/4/3 (Phase 1/2/LoRA) units; the authors' registered "at least 6/9" HELD for ridge and Phase 1 Procrustes, and FAILED for Phase 2 and LoRA Procrustes.

    Attacks and canary (Section 5.8)

    • A1/A1m: A1 keeps AUROC unchanged and the AUC rule does not alarm (P-A1.1, P-A1.2, HELD); the two-moment model predicts accurately in only 0.656 of units (P-A1.3, FAILED), and the estimated-rate check has sensitivity 0.951 and specificity 0.568 (P-A1.4, FAILED).
    • A2: Across 30 units, overall AUROC shows no drop ≥0.05 (P-A2.1, HELD), and overall FPR stays within ×1.5 (P-A2.4, HELD); cat-3 reaches stealthy (c = −2) in 3 of 5 seeds, REFUTING Hypothesis A2 that "no single attacker evades both checks simultaneously"; the estimated-rate check flags 7/30, more than predicted (P-A2.2, FAILED).
    • Canary: The post hoc canary flags all 8 (9 by frozen-threshold reading) blinded units with c = −2, with no false alarms across 603 clean splits (guaranteed by construction of its threshold rule); the authors note the canary was fixed after A2 scoring, and no attacker adapted to it.

    Other ablations and analyses

    • LoRA dose (Table 4): The frozen error of the safety-corpus arm is 5.39–7.00 times that of the general arm (descriptive); the registered prediction that PKU-SafeRLHF damage exceeds the general arm HELD on the fixed split; damage does not grow with dose, H-LoRA-1 NOT SUPPORTED.
    • MLP probe (Section 5.6): 7 of 9 Phase 1 units have paired ratio ≥ 2 (P-MLP.1, HELD), and the ratio difference from the linear probe is within ×1.5; failure and repair are not limited to linear probes.
    • XSTest over-flagging (Section 5.4): In Phase 2, after the 2d update, safe prompts are flagged 2.178 times [1.867, 2.733], and 2a is 0.265 times [0.025, 0.086]; for Phase 1's 1a at L16, the paired ratio is 9.43, and the mean share of the 250 safe prompts flagged rises from 0.053 to 0.480.
    • Hardware control (Section 4.2): The paired ratio for the V100/A100 control is within 0.9995–1.0029, but activations are bitwise identical in only 1/17 layers, and registered row 13 FAILED; the LoRA series was subsequently re-measured on A100, with no registered verdict changing.
    • CKA (Section 5.1): The minimum CKA per layer for 1a–1h is no lower than 0.855; the authors state that representational similarity is not a reliable guide to damage.
    • Registered records (Section 6.2): Of 137 registered entries, 69 HELD, 24 FAILED, 5 NOT SUPPORTED, 3 REFUTED, 6 withdrawn/undefined/uncertain, and 30 outcome-only or descriptive.
  5. What are some points that can be further explored?

    The authors list limitations including pairing selection, confounding, data exposure, and model scope; the experiments cover 7B–9B open-source models and two safety datasets.

    Limitations and future directions noted by the authors

    • No registered selection rule for Phase 1 pairs: Section 6.2 notes that the 7 Phase 1 pairs predate the registration rule, no existing written rule can derive them, and 8 of their 9 headline units fail closed; Phase 2 is fixed by the registration rule, with 3/9 failing.
    • Confounding of phase, update type, and readout: Section 6.2 notes that all Phase 1 pairs start from base models and read raw prompts, while Phase 2 involves instruct-side updates and reads the chat template, so the three are aliased; for 2b, 2c, and 2e the rendered input changes with the weights, and the authors did not separate the two.
    • Configuration changes not separated: Section 6.2 notes that 1e, 1g, 2a, 2b, 2d, and 1h, 2j change model configuration (Table 2) and were not separated from weight changes.
    • HH-harmless exposure: Sections 6.2 and 5.7 note that the training dialogues of this arm contain evaluation items in other user turns (810 dialogues and 763 distinct items at 400 steps), and no record defines paraphrase or near-duplicate contamination; the authors therefore use PKU-SafeRLHF as the clean safety arm.
    • Model and data scope: Section 6.2 states that only open-weight models of up to 9B parameters were studied; the controlled LoRA uses one checkpoint (Mistral-7B v0.1), and the model-side attack uses one backbone (Mistral-7B-Instruct-v0.2); probes are trained only on Aegis 2.0 and WildGuardMix, and over-flagging is measured only on XSTest.
    • Adaptive attackers out of scope: Section 6.2 notes that no attack adapted to the canary (the canary was fixed after A2 scoring); Section 6.2 also notes that "no stale units" depends on a margin of only 0.003 between −0.0470 and the big drop line, and that only one headline unit fails open, so this cannot be generalized.

    Experimental coverage

    • The updates tested are 16 public checkpoint pairs (18 headline units) plus 1 supplementary pair 1h and 1 non-headline pair 2j, all 7B–9B open-weight models (Section 4.2, Table S-I).
    • Probes are linear logistic regression, with one one-hidden-layer MLP probe re-checked on 9 Phase 1 units (Section 5.6); 5 seeds and 1000 resplits per unit.
    • Defense/repair evaluation covers threshold resetting (Rsame, Rfresh), quantile transfer, restandardisation, and ridge/Procrustes realignment; poisoning experiments are run only on a reference set of n = 1000 (Section 5.5).
    • Attacks cover A1 (simulated shift), A1m (rank-1 LoRA), A2 (category-targeted, 3 categories × 2 c values × 5 seeds = 30 models), and A3 (36-unit reference-set poisoning); the canary is post hoc, and adaptive attacks were not tested (Sections 4.6, 5.8).
    • The paper reports no review model (all verdicts rely on threshold rules and AUROC, with no LLM or human review); partial input caches for 4 headline units have no extraction record, and the same-GPU assumption relies on one validation record read after the fact (Section 6.2, Supplement S-7).
  6. Summarize the main content of the paper

    Frozen probe thresholds shift substantially in operating point after backbone updates while AUROC does not move; the authors provide label-free detection and repair, and demonstrate update-controlled attacks.

    An analysis paper on the operation and maintenance of activation probe guardrails: after the deployer sets the threshold at a 1% false positive rate and freezes it, the backbone model is replaced by a released update, and the question is whether the probe's operating point still holds.

    • Problem: AUROC-type staleness checks (such as Duan [1]'s standard) do not read the threshold and cannot answer whether the FPR/TPR at the frozen threshold survives; the authors call this a different question from AUROC.
    • Method: On 18 headline units from 16 public checkpoint pairs, with 5 probe seeds and 1000 resplits, measure the paired ratio; use a two-moment model to explain the change, an estimated-rate check for label-free detection, and label-free methods such as threshold resetting for repair, and record all verdicts with 137 registered predictions.
    • Main result 1: At the 1% target, 8 of 9 Phase 1 units fail closed, and in Phase 2, 2 close and 1 opens; the largest increase is Mistral-7B v0.1→Instruct on Aegis 2.0, with FPR going from 1.01% to 15.12%, while across the 18 units the AUROC change does not exceed 0.0058, and the Duan standard does not judge any unit stale (Table 3, descriptive).
    • Main result 2: The two-moment model explains the change to within ×1.5 for 0.905 of the 63 Phase 2 registered units (HELD); the estimated-rate check flags 0.918 of damaged Phase 1 layer units and passes 0.898 of undamaged units, whereas the AUC rule flags only 0.019.
    • Main result 3: Resetting the threshold on stored or fresh benign prompts returns FPR/target to 0.96–1.02 for the 9 Phase 1 units without new labels; but with a 1% poisoned 1000-item reference set, resetting flags only 0.006–0.017 of harmful prompts (descriptive).
    • Main result 4: An update-controlling attacker (A2) achieves stealthy blinding on cat-3 in 3 of 5 seeds, with both checks silent, and the registered Hypothesis A2 is REFUTED; a post hoc canary flags all such units.
    • Authors' conclusion: After each update, deployers should re-measure the FPR at T using stored benign prompts passed through the updated model, at the target rate in use, and reset the threshold on the same batch of prompts if necessary; the authors emphasize that the reference set must be placed where the attacker cannot write, and that realignment preserves AUROC but cannot reliably restore the operating point.

Related papers

Show 3 more papers
Read originalarxiv.org