Counterfactual Neighborhood Risk Field: Evidence-Driven Selective Safety for Vision-Language Models

1. 01What problem does this paper attempt to solve?

   The authors believe that representation separability is insufficient to decide when to trust risk scores, so CNRF instead uses selective evidence.

   The authors believe representation separation cannot explain what risk scores mean or when they are trustworthy, so CNRF does selective intervention accordingly.

   • Scenario: The authors say a VLM must both assist and prevent harmful capability activation. Harmful and benign requests can share topics, entities, and images; over-refusal reduces utility. Typography, distracting images, and cross-modal combinations change the representation of intent.
   • Existing shortcomings: Representation defenses often do not modify the victim model, but learn to separate two classes of inputs and score them. The authors believe separation cannot explain what the score represents or when the evidence is trustworthy; the score may mix harmful capability with unfamiliarity or insufficient coverage. In Figure 1, an overly broad boundary may let harmful assistance through, and an overly strict boundary may reject benign requests with risky context.
   • Observation: The authors believe the key is not whether the reference is global or local; RCS already uses class-relative distances and neighborhood comparisons to distinguish maliciousness from novelty. Context-matched counterfactual pairs change the requested capability and preserve surrounding content; their representation difference gives a capability-related direction; neighborhood support separately measures whether the evidence applies to that query.
   • Method: Propose CNRF. At inference time, use counterfactual directions and a neighborhood risk field to estimate risk, with support separated from risk; after routing through a victim-model-adaptive interface, the same model reviews, releasing acceptable candidates or generating constrained alternatives with lower harmful capability.
   • Threat model
     • Goal: Identify harmful capability requests and preserve appropriate responses under benign variants.
     • Knowledge: The paper does not state the attacker's knowledge level. The defense reads the victim VLM's intermediate representations, and the observation stage does not generate task restatements or candidate responses.
     • Actions: Evaluation inputs come from five published multimodal jailbreak families. The defense uses an offline counterfactual library, does review or constrained generation on routed requests, and does not modify model parameters.
     • Victim systems: Qwen2.5-VL-7B, Gemma-3-12B, InternVL3-8B. Each victim uses the same fixed configuration across all attack families and benign queues.

   Collapse this answer

2. 02What related research exists?

   The authors place the difference between CNRF and RCS in the evidence relationship, not in local versus global.

   Section 2 of the paper divides related work into three groups. The introduction uses jailbreak and over-refusal literature to illustrate the problem, without criticizing those works as method baselines.

   • Representation safety evidence: Jiang et al. (2025)'s HiddenDetect identifies intrinsic safety activations from hidden states; Fang et al. (2026)'s VLMGuard learns safety representations from unlabeled mixtures; Nian et al. (2025)'s JailDAM uses adaptive unsafe knowledge memory for detection. The paper describes them as extracting inference-time risk signals from VLM states.
   • RCS: Hua et al. (2026) uses class-relative representation distances to handle the confusion between novelty and maliciousness; KCD uses local nearest neighbors, MCD fits Gaussian class distributions. The paper says this is the work closest to CNRF, and states the difference is not whether the information is local or global, but the relationship preserved by the reference: CNRF takes counterfactual capability transfer matched to the context, preserves endpoint correspondence in the risk coordinate, and separately estimates neighborhood support. The paper also connects this perspective to counterfactual augmentation by Kaushik et al. (2020) and selective prediction by Geifman and El-Yaniv (2019).
   • Input filtering and response-level defenses: Wang et al. (2024)'s AdaShield uses defense prompts to mitigate jailbreaks. Xu et al. (2024)'s CIDER and Zhang et al. (2025)'s JailGuard evaluate defenses through input perturbation and stability analysis. Gou et al. (2024)'s ECSO checks answers before visual-to-text regeneration; Ding et al. (2025)'s ETA evaluates input and response first, then applies safety prefixes and sentence-level alignment. The paper says these works show interventions should not just be refusal; in contrast, CNRF first does evidence screening, then the same victim model reviews candidates.
   • Risk-aware internal interventions: Wang et al. (2025), Xia et al. (2025), Fu et al. (2026)'s ASTRA, OH-Defense, CARE intervene through activation editing, attention manipulation, or safety subspace modification. Park et al. (2026)'s MoRAS scales steering by estimated risk; Wang et al. (2026a)'s RAI enhances unsafe prototypes in visual tokens; Chen et al. (2025)'s SafePTR combines pruning with feature recovery. Liu et al. (2026)'s SafeCoDe contrasts real and perturbed visual inputs and adjusts refusal token probabilities. The paper says they all do risk-aware intervention like CNRF, but CNRF does not directly modify the victim model.
   • Baselines and benchmarks: Experimental comparisons are HiddenDetect, RCS-KCD, RCS-MCD, VLMGuard, and no defense, AdaShield-A, ECSO, CIDER, ASTRA, OH-Defense, CARE. Attack families are JOOD, JailBreakV, UltraBreak, HADES, MMJailBench; benign utility uses MOSSBench, MM-Vet, MMMU. Counterfactual construction uses Alpaca, VizWiz, AdvBench, MM-SafetyBench. JailDAM, ETA, JailGuard, MoRAS, RAI, SafePTR, SafeCoDe appear in related work and are not included in the above experimental baselines.

   The authors position this paper as: not directly learning the harmfulness boundary, but constructing applicable evidence and separating evidence applicability from risk estimation, thereby doing selective routing and response review without modifying the model.

   Collapse this answer

3. 03How does the paper address this problem?

   Counterfactual directions estimate capability risk, support determines whether evidence applies, then selective review.

   CNRF (Counterfactual Neighborhood Risk Fields) constructs evidence at inference time: context-matched counterfactual pairs provide capability directions, the neighborhood field estimates risk, support determines whether the evidence covers the query, and then routed requests undergo response review by the same victim model. It does not modify victim model parameters.

   Counterfactual evidence and capability directions

   • Pairing: For harmful requests, construct benign counterparts that change the requested capability while preserving topic, entities, language form, and visual dependence as much as possible. Multiple presentations of the same parent are used to separate capability changes from presentation changes. Local Gemma-3-12B handles rewriting, and representations are extracted from the victim model. Appendix A.1 gives filtering conditions and generation settings.
   • Observation and projection: Prefill observation reads normalized hidden states, can use the original request, a fixed safety verbalizer, or a combination of both; the observation stage does not generate task restatements or candidate responses. A trained fitted projection yields query coordinates z, preserving paired capability differences and compressing irrelevant changes. Fitting, calibration, and validation are separated by parent and shared image groups to avoid leakage.
   • Direction: The midpoint of benign endpoint bi and harmful endpoint qi, direction is m_i=(b_i+q_i)/2, a_i=q_i−b_i. The midpoint anchors the comparison in the observed context, and the direction is the capability-related displacement under that context. Projection along that direction normalizes both endpoints to -1 and +1 to remove scale differences.

   Neighborhood risk field and support

   • Aggregation: The nearest k different-parent midpoints in projected Euclidean space form the neighborhood. Non-negative weights βi aggregate directions to get the query-relevant direction; risk is the projection along that direction minus the intercept. Qwen's intercept uses paired midpoints, InternVL uses local benign references. βi is fitted by parent-weighted logistic regression, shrunk toward 1, and excludes the query's own parent. Fitting benign statistics then standardizes the raw risk into a signed score r.
   • Support: s is normalized from the distance to the k-th different-parent benign neighbor; larger means weaker coverage. It measures evidence applicability, not harmfulness. The authors say separating the two avoids treating unfamiliarity as harmfulness.
   • Routing: When internal safety observations exist, the log-probability difference over harmful and benign verbalizer sets can be used, then averaged over selected layers. If priority ψ exceeds a threshold, route. Qwen's ψ equals r, and support state does not independently trigger review. InternVL's ψ is min{max(r, s), r+1}, and support elevation does not exceed one fitted benign risk interquartile range. When the direction norm does not exceed 10^-12, representation editing is not authorized; only support exceeding the threshold routes.

   Victim-model-adaptive implementation

   • Readers: Qwen and InternVL use local readers, Gemma uses an internal safety observer. Table 3: Qwen is verbalizer observation, 8 normalized blocks, projection rank 8, k=16, ridge 0.01, shrinkage 0.01, benign budget α=0.15; InternVL is native plus verbalizer, 16 blocks, rank 4, k=4, ridge 1, shrinkage 0.1, α=0.20. Both read the last token and non-image token mean at relative depths 1/4, 1/2, 3/4, 1.
   • Thresholds: Risk or priority cutoff approximately 1.020079 (Qwen) and 1.050150 (InternVL); support cutoff approximately 1.447632 and 1.368194. Calibration uses training-role data, run budget is selected from development results, and final evaluation remains fixed. Gemma averages safety scores over the last token of blocks 19–26, uses the original input and positive score direction, threshold approximately 0.328594. The local field does not participate in its screening; counterfactual references still enter the separately calibrated response stage.

   Response review by the same victim model

   • Release rule: Routed requests first generate buffered candidates. Response verification plus retrieved counterfactual cards (capability axes and allowed alternatives) decide whether to release. If not routed or release conditions are met, output the original candidate; otherwise, generate an alternative with lower harmful capability according to compiled constraints. The detection library and response evidence library are calibrated separately. Full training examples are not written into response instructions.
   • InternVL additional rule: If verification attributes refusal to insufficient coverage, and there is no hard alert or positive safety evidence, then do one more same-model check; the capability score must be below 3. Alerts, malformed checks, or scores of at least 3 veto release. Qwen and Gemma do not have this parser.
   • Visual path: Visual fallback generates omitting the original image and source instruction, and adds a fixed refusal prefix; ordinary answers and verified releases retain original visual information. Appendix A.4 gives review prompts for attack harm, MOSS refusal, and benign answer classification.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Three-model macro DSR is 88.57%, 86.06%, 89.30%; the authors say each model's SUS point estimate is highest.

   Experimental setup

   • Victim models and data: Qwen2.5-VL-7B, Gemma-3-12B, InternVL3-8B. Table 2: 3,159 per victim. Attacks are JOOD 510 (170 targets, best of three), JailBreakV 280, UltraBreak 315, HADES 750, MMJailBench 544. Benign are MOSSBench 300, MM-Vet 160, MMMU 300. Each victim uses the same fixed CNRF configuration on all queues.
   • Baselines: Representation screening is HiddenDetect, RCS-KCD, RCS-MCD, VLMGuard. End-to-end is no defense, AdaShield-A, ECSO, CIDER, ASTRA, OH-Defense, CARE. Appendix B.1 states that the two RCS variants are fitted on CNRF counterfactual archives, evaluating the scoring procedure under shared supervision, not reproducing original RCS benchmark numbers; implementation uses k=40, the original paper writes k=50.
   • Metrics and review: Screening reports AUROC, TPR at 5% FPR, standardized pAUC for FPR 0–10%, attack routing rate, and benign pass-through rate. End-to-end ASR is the proportion of harmful assistance within a family, DSR is 1 minus ASR, macro ASR is equal weight across five families. Local Gemma-3-12B gives harm scores 1–5, s≥4 is successful harmful assistance; also define ASR3 with s≥3, SUS still uses the main criterion. SUS is 100 times the equal-weight average of defense success and benign utility; benign utility averages MM-Vet, MMMU accuracy, and 1 minus MOSS refusal rate. Table 1's MOSS column is 100 minus F_MOSS.
   • Decoding and judgment: All methods share inputs, bfloat16, greedy decoding, up to 256 new tokens. Attack review sees images. MM-Vet review does not receive victim or defense identifiers, partial scores 0 to 1 in steps of 0.1, greedy, single, 16 output tokens; the paper states this differs from the official GPT-4 protocol. Unknown results keep the original denominator. One InternVL CIDER judgment was unresolved and conservatively counted; the main CNRF judgment is complete. This review model shares a name with one of the victims.
   • Routing protocol: For detection, JOOD takes the lowest score of three attempts, and routes a target only if all three are routed; end-to-end JOOD uses best-of-three. ROC operating points are only for evaluation; deployment cutoffs are chosen separately. The paper does not report repetition counts for the main panel.
   • Other fixed settings: Counterfactual library uses Alpaca, VizWiz, AdvBench, MM-SafetyBench. Appendix A.1: UltraBreak images are transferred from Qwen2-VL-7B, no victim-specific optimization. Thresholds are determined on training-role or development-role data and then fixed.

   Main results

   macro DSR is from §4.2, the rest from Table 1's CNRF row. SUS and utility columns are Table 1's percentage scale.

   Swipe horizontally to read the table

   Victim	macro DSR	SUS	Native SUS	MM-Vet	MOSS non-refusal
   Qwen2.5-VL-7B	88.57%	69.82	60.24	63.25	58.67
   Gemma-3-12B	86.06%	73.27	64.54	67.88	77.00
   InternVL3-8B	89.30%	77.21	70.88	75.63	70.33

   • The authors say among the evaluated defenses, CNRF's macro DSR is highest on Qwen and Gemma; on InternVL it is about 0.12 points lower than OH-Defense, but benign task utility is better. In Table 1, OH-Defense on InternVL has MOSS non-refusal, MM-Vet, MMMU of 26.33, 5.75, 30.00.
   • The authors say MM-Vet drops about 4.94, 0.00, 2.88 points relative to native, and MMMU differs from corresponding native by no more than 2.67 points. Table 1 native MM-Vet is 68.19, 67.88, 78.50, native MMMU is 52.67, 41.33, 56.00, CNRF's MMMU is 50.00, 41.00, 56.00. Qwen native MOSS non-refusal is 72.00, ECSO is also 72.00.
   • The authors say SUS point estimates are highest among evaluated defenses on all three victims; on InternVL, CARE is 76.68. The lowest attack-family DSRs named in the text are Qwen's MMJailBench 76.10% and Gemma's JOOD 77.65%. On several families other defenses are higher: on Qwen's MMJailBench, OH-Defense is 95.22 and CARE is 92.46; on Gemma and InternVL's UltraBreak, OH-Defense is 100.00 on both, and on InternVL CARE is 99.37. Appendix Table 4's DSR and utility differ from Table 1; C.1 says the protocol is the same, and the paper does not explain the difference; here Table 1 and §4.2 take precedence.

   Representation screening

   • What was measured: Figure 3 and Table 5 compare continuous ranking and deployment routing on the same 3,159. CNRF's macro AUROC, TPR@5% FPR, attack routing, MOSS pass-through, MMMU pass-through are: Qwen 87.87%, 60.76%, 74.55%, 79.33%, 94.00%; Gemma 94.50%, 80.75%, 72.44%, 89.00%, 99.67%; InternVL 92.74%, 74.93%, 81.41%, 84.67%, 100.00%. MM-Vet pass-through is 92.50%, 100.00%, 96.25%.
   • Comparisons: The authors say CNRF's macro AUROC exceeds evaluated baselines on each victim. HiddenDetect on Qwen has MOSS pass-through 90.33%, but attack routing only 39.84%. Figure 3 notes pass-through measures acceptance, not answer correctness.
   • Reader differences: Appendix D.2 states that removing the local field on Gemma leaves routing unchanged on 3,159; restoring the local field without the observer changes 1,254. The authors say Gemma's detection results characterize observer screening, not the local coordinate geometry of Qwen and InternVL.

   Evidence quality ablations

   • What was measured: §4.4 changes only evidence construction on a fixed Qwen reader. Figure 4 and Table 6 use 770 validation inputs. Matched, global, random, far-neighbor pAUROC are 89.24, 85.92, 75.24, 48.59. The authors say the matched-vs-global gap of 3.32 points comes from query-conditioned references.
   • Direction correspondence: Direction mismatch drops pAUROC from 89.24 to 75.32, AUROC from 94.14% to 77.25%. On paired endpoints, matched AUROC is 94.58%, mismatched is 91.12%. The authors say effective evidence depends not only on neighborhood proximity but also on context similarity and capability transfer correspondence.
   • Conditions: Table 6's random and direction mismatch are averaged over ten seeds. Table 8 separately writes 770 held-out training-role inputs; the paper does not state whether this is the same set as the above 770 validation inputs.

   Response review and overhead

   • Pilot release: Figure 5(a) and §4.5 look at a Qwen pilot under fixed screening: 130 routed stratified inputs, 16 per benign benchmark. Of 48 benign inputs, 12 are newly released; MOSS, MM-Vet, MMMU are 2/16, 4/16, 6/16. macro DSR stays 75.42%. MOSS non-refusal goes from 56.25% to 62.50%, MM-Vet partial correctness from 35.00 to 58.13, MMMU from 37.50% to 50.00%. The figure note states bars count released candidates different from the frozen response, not new correct answers.
   • Author framing: Appendix D.3 writes macro ASR stays 24.58%, SUS goes from 55.65 to 65.56; card-free review improves pilot SUS by 9.91 points over the existing response, paired 95% bootstrap interval [3.62, 19.51]. The authors state this pilot does not establish card-specific advantage or estimate performance on the full panel.
   • Overhead: Figure 5(b) measures 32 benign MM-Vet on the same GPU. Average latency (seconds) and peak memory (GiB): Qwen native 1.587/16.58, ECSO 1.925/16.60, CNRF 1.866/17.21; Gemma 3.265/23.05, 3.658/23.12, 3.585/23.07; InternVL 1.201/15.53, 1.662/15.58, 1.752/16.05. The authors say relative to native, latency increases 0.279–0.551 seconds, peak memory increases at most about 0.02–0.64 GiB; relative to ECSO, Qwen and Gemma are lower by 0.059 and 0.073 seconds, InternVL higher by 0.090 seconds. This benign subset does not trigger ECSO's conditional path.

   Other ablations and analyses

   • Table 7: On 770, matched-minus-random AUROC difference is 8.09 percentage points, interval [4.99, 11.53]; matched-minus-global is 0.85, interval [-1.17, 2.67], and the authors say this interval crosses 0.
   • Table 8: Qwen matched AUROC 94.14, equal-weight β is 78.54, reassignment is 66.54–87.77; InternVL matched 87.17, novelty 95.54, global 76.78, reassignment 86.25–89.36.
   • Table 9: On 430 merged dual-label carriers, Qwen matched 94.58, global 94.00, novelty 68.84; InternVL matched 86.60, global 91.53, novelty 81.54. The authors say novelty is highly predictive when benign-only carriers are included, lower on the paired subset.
   • Tables 10–11: On InternVL's full 3,159, full review relative to generic review has macro ASR 0.77 percentage points higher, interval [0.34, 1.27], MOSS refusal failure 4.33 percentage points lower; MM-Vet and MMMU differences are 0. The authors say Qwen and Gemma show no differences on these metrics.
   • Appendix E: From InternVL's full review, 8 of the 13 additionally released MOSS are selected. All are routed requests, and released text matches the buffered native answer. The authors say these are examples selected by outcome, not representing frequency, human correctness, or the separate effect of counterfactual cards.
   • Appendix Table 4 gives another set of end-to-end numbers, e.g., Qwen native JOOD DSR is 74.12, Table 1 is 44.12; the paper does not explain the difference.

   Collapse this answer

5. 05What are some points that can be further explored?

   Section 5 does not list limitations item by item; the appendix requires independent prospective evaluation and hints at automated review bias.

   Limitations and future directions noted by the authors

   • Section 5: Titled Conclusion and Limitations, the body does not list limitations or future experiments item by item. The section only states that these findings point from boundary learning toward interpretable evidence construction.
   • Appendix A.1: The evaluation characterizes performance on the selected benchmark suite; independently sampled prospective evaluation remains a separate requirement.
   • Appendix A.2: Automated judgment may inherit shared model-family biases and should be interpreted separately from human evaluation.

   Experimental coverage

   • The victims tested are Qwen2.5-VL-7B, Gemma-3-12B, and InternVL3-8B. Table 2 safety queues are JOOD 510, JailBreakV 280, UltraBreak 315, HADES 750, MMJailBench 544; benign are MOSSBench 300, MM-Vet 160, MMMU 300; 3,159 items per victim. One fixed configuration per victim.
   • End-to-end comparisons are no defense, AdaShield-A, ECSO, CIDER, ASTRA, OH-Defense, CARE; representation screening comparisons are HiddenDetect, RCS-KCD, RCS-MCD, VLMGuard. Decoding is bfloat16, greedy, up to 256 new tokens. The paper does not report replicates for the main surface.
   • Harmful assistance is scored 1–5 by a local Gemma-3-12B, with the primary criterion s≥4 and an additional ASR3 defined as s≥3. MM-Vet is scored by the same local model in 0.1 increments. The paper does not report agreement between automated review and human evaluation.
   • Qwen and InternVL use a local reader (Table 3: k=16 and 4, α=0.15 and 0.20). Gemma screening uses an internal safety observer from layers 19–26 with a threshold of about 0.328594. The Qwen reference-selection experiment uses 770 validation inputs; Table 8 also writes 770 held-out training-role inputs. The paper does not state whether these are the same set.
   • Response-action comparisons are on InternVL's 3,159 items across native, fixed refusal, generic review, and full review. Qwen additionally has a 130-item routed stratification pilot. Latency and memory are measured on 32 benign MM-Vet items per victim. The appendix reports bootstrap intervals for some comparisons.

   Collapse this answer

6. 06Summarize the main content of the paper

   Three-model macro DSR is 86.06%–89.30%, and the authors say evidence applicability should be separated from risk.

   CNRF is an inference-time selective safety framework for VLMs that does not modify the victim model's parameters.

   It addresses the problem that harmful and benign requests can share topics, entities, and visual context. The authors state that representational separability does not determine what a risk score means or when it is trustworthy; a score that treats unfamiliarity or under-coverage as harmfulness can either allow harmful assistance or over-refuse.

   Methodologically, CNRF constructs context-matched benign controls for harmful requests, uses representation differences as capability directions, and aggregates them over neighbors into a query-relevant risk field; support separately measures whether these references cover the query. Qwen and InternVL use a local reader, while Gemma's screening instead uses an internal safety-token observer. Routed requests are handled by the same victim model generating candidates: if verification passes, the original is released; otherwise, constrained generation is performed along retrieved capability axes.

   The authors report macro DSR of 88.57%, 86.06%, 89.30% across five jailbreaks on Qwen2.5-VL-7B, Gemma-3-12B, and InternVL3-8B, with SUS of 69.82, 73.27, 77.21, and state this is the highest SUS point estimate among the defenses evaluated on each victim. Representation screening macro AUROC is 87.87%, 94.50%, 92.74% (Table 5). On the Qwen validation set, matched-neighbor pAUROC is 89.24 versus 48.59 for far neighbors, and direction mismatch drops AUROC from 94.14% to 77.25% (Figure 4, Table 6).

   The authors argue that reliable evidence depends on query-matched references and correspondence between context and capability direction, so representation safety should move from boundary learning to applicable evidence. The appendix also states that on InternVL, full review raises macro ASR by 0.77 percentage points relative to generic review; on Gemma, the screening results characterize the observer, and removing the local field leaves 3,159 routed items unchanged.

   Collapse this answer

Related papers