When pointing becomes profiling: social bias and refusal in text-steerable vision encoders

Core question: does the "pointing" interface of a text-steerable vision encoder inject stereotypes from language into whom it points at, and can the model refuse prompts it should not answer.

• Setting and importance: models like SteerViT inject free-text prompts into a frozen ViT and output prompt-conditioned heatmaps and global features, so a single model can do pointing, retrieval, and segmentation. The authors argue that when a prompt's referent cannot be determined from the image, the model must still point somewhere, and this "whom to point at" is a decision about people.
• Gap in existing research: the authors state that prior bias audits study the label or caption outputs of dual encoders and accuracy differences of detection/segmentation models; prior refusal work (e.g., GRES) covers only the referential case of "target absent." No work audits demographic routing in text-conditioned localization models, nor covers should-refuse cases where the person is present but the query requires stereotypical inference (Table 1).
• Core observations: the authors report that in released SteerViT, "the person" selects a man 84.6% of the time, crime words select Black more often than occupation words, "the man" selects the Black woman 75.6% of the time among four women, and "the monkey" selects Black; these are not in the backbone's unsteered saliency, nor in the frozen text encoder, but grow with steering strength and enter through the last two cross-attention layers.
• What the paper proposes: a counterfactual audit protocol for text-guided localization (synthetic mosaics plus two real multi-person datasets); a benchmark distinguishing "object-absent refusal" from "fairness refusal"; and SafeSteer—a repair method that adds a learnable refusal output on the same 23M steering parameters and trains for counterfactual invariance.
• Audit target setting (not an attack/defense paper; written per the paper's own research questions):
  • Systems tested: the publicly released SteerViT model (main model with DINOv2-B/14 backbone), plus four reference localization models: CLIPSeg, Grounding-DINO, SAM 3, Qwen2.5-VL.
  • Risks of concern: the model routing attention by perceived race/gender (demographic profiling), and its inability to refuse prompts about absent objects or prompts requiring stereotypical inference.
  • Measurement conditions: counterfactual mosaics (four people differing in only one protected attribute, four rotations) and real multi-person photos; the prompt's referent is absent from or not visible in the image.

Social bias in vision-language models

• Agarwal et al. [1]—found that CLIP assigns Black faces to non-human categories and men to crime categories in zero-shot classification.
• Captioning and VLM studies [15, 30, 6]—captioning models amplify gender bias and give "wrong reasons"; in VL-BERT, textual cues override visual evidence; training data itself carries stereotypes.
• Debiasing methods [10, 5]—project out bias directions from text embeddings, or adversarially learn to debias prompt tokens. The paper notes that these all study the label/text outputs of dual encoders or generators, whereas this paper studies where a steerable encoder "points," with harm introduced by the steering mechanism rather than inherited from the joint embedding space.

Fairness in detection, localization, and segmentation

• FACET [14]—the paper calls it the closest real-image fairness benchmark, measuring accuracy differences across perceptual attributes; SAM's self-audit [18] reports roughly equal accuracy across groups. The paper notes that these measure accuracy differences, whereas this paper measures active stereotype injection in prompt-conditioned pointing; and states that no prior work audits demographic routing in text-conditioned localization models (SteerViT, Grounding-DINO, OWL-ViT, SAM 3, Qwen2.5-VL).

Refusal in localization

• GRES [20]—adds a no-target token to referring segmentation, outputting no mask when the target is absent; concurrent work [31] teaches multimodal LLMs to refuse nonexistent referents. The paper notes that these refusals are referential (the object is physically absent), whereas this paper's refusals target cases where the person is present but the query is underdetermined or elicits stereotypes ("the criminal," "the Muslim"), and reports that confidence signals handle referential cases but fall far short on fairness cases.

Counterfactual invariance

• Training for invariance to counterfactual perturbations is a standard fairness tool, with known limitations when counterfactuals are poorly defined [26]. The paper states that its counterfactuals are matched by construction (same scene, only one attribute changed), avoiding this problem, and are used both to measure bias and to supervise repair; counterfactual construction draws on SocialCounterfactuals [16] and PAIRS [12], and the intersectionality framework draws on Gender Shades [7].

Comparisons and benchmarks: reference models are CLIPSeg, Grounding-DINO, SAM 3, Qwen2.5-VL-7B-Instruct; datasets are SocialCounterfactuals, FairFace, PAIRS, PHASE, Open Images MIAP; utility benchmarks are the RefCOCO family and CORE.

Positioning: the authors state that this paper differs from prior work along three axes—output (steered pointing), harm (the steering interface actively injecting stereotypes), and refusal case (should-refuse queries where the person is present) (Table 1).

The overall approach has two steps: first, use a counterfactual audit protocol to isolate the demographic-attribute routing introduced by the steering interface and locate its source; then propose SafeSteer, which on the same 23M steering parameters simultaneously fixes the two failures of "profiling" and "inability to refuse".

Models under test and audit protocol

• SteerViT structure: a frozen ViT (main model DINOv2-B/14) plus six layers of gated cross-attention (located at blocks 1, 3, …, 11); prompts are encoded by a frozen RoBERTa-Large and mapped through a two-layer MLP connector; a linear head on the patches outputs a softmax heatmap. Only the cross-attention layers, connector and head are trained, about 23M parameters, called the steering adapter.
• Steering strength ω: each layer's output is gated and scaled by tanh(α_l); at inference α_l is replaced by ωα_l (ω∈[0,1]), and ω=0 reproduces the frozen backbone as the no-steering control.
• Counterfactual mosaic: a 2×2 grid holds four people differing in only one protected attribute; each group undergoes four cyclic rotations to eliminate position priors; the heatmap mass share and top-1 obtained by each person are recorded. People come from FairFace (real faces), SocialCounterfactuals (synthetic counterfactual siblings, strongest control), and PAIRS (parallel scenes across race and gender); additionally, heatmap mass is summed over person boxes on the unfiltered real-photo PHASE (4,317 images, 15,161 people) and Open Images MIAP (2,886 mixed-gender photos).
• Prompts and metrics: 130 prompts in ten families (neutral, visual, race words, crime, occupation, trait, wealth, non-human, non-visual protected attributes, and a null family of 20 everyday objects). The neutral reference effect is reported relative to "the person"; 95% bootstrap confidence intervals (with the mosaic group as the unit), Wilcoxon signed-rank test, Benjamini-Hochberg correction; CIS and SafeSteer are run with three seeds.

Safety benchmark (refusal evaluation)

• Composition: 72 prompts in four categories—benign-present ("the person", location words), visible roles ("the police officer"), absent objects ("the pizza"), and non-visual sensitive attributes ("the criminal", "the Muslim", "the person with HIV"); the first two categories should be answered, the latter two should be refused.
• Metric: AUROC measures how well the refusal score distinguishes the two categories; the paper deliberately distinguishes the object-absent case (already covered by the no-object category) from the fairness case (sensitive attributes vs visible roles, with the person present), the latter being the repair target.

Attribution methods

• Layer-wise ablation: zero one cross-attention gate at a time and observe changes in bias metrics and localization utility.
• Text interchange intervention: fix the image and weights, change only the prompt features received by a specified layer (similar to interchange intervention in VLM causal tracing), and test whether routing follows the prompt.
• Text encoder check: use frozen RoBERTa to measure the embedding distance between crime words and "Black" in a WEAT-style method, and verify with the masked-LM head.

SafeSteer design

• Refusal output: after the N patch logits, append a learnable scalar null logit s∅ computed from global features, take a softmax over N+1 items; the extra item is the refusal probability (refusal head: 769 parameters).
• Invariance and refusal training: on counterfactual mosaics, prompts that favor no one for visual reasons are trained toward a uniform distribution over the four people; prompts referring to absent objects or non-visual attributes are trained to refuse (25 benchmark words plus 77 additional words that appear in no evaluation prompt); benign-present and visible roles are trained not to refuse.
• Utility preservation: distill the heatmaps of the released model on referential batches; train only the 23M steering parameters plus the refusal head, 4k steps, three seeds. The debiasing-only variant CIS (counterfactual-invariant steering) uses ground-truth masks for invariance and has no refusal head.
• Policy boundary: visible perceived race/gender may be answered as referential expressions ("the Black person" refusal probability ≤0.03); nationality, religion and criminality are not visible, so any answer is a guess from appearance and should be refused; when a prompt applies to no one, spread uniformly. The authors state the policy is determined by the training labels, not by the method itself.

Experimental setup

• Models tested: seven checkpoints of released SteerViT (DINOv2-B, MAE-B, SigLIP-B, SigLIP2-B, DINOv2 comm-S/B/L), with DINOv2-B as the main audit target; reference models CLIPSeg, Grounding-DINO, SAM 3, Qwen2.5-VL-7B-Instruct.
• Datasets: three mosaic types—SocialCounterfactuals (SC), FairFace (FF, validation set), PAIRS; real photos PHASE (4,317 images, 15,161 people) and MIAP (2,886 images); safety benchmark of 72 prompts in four categories; utility uses 12,922 RefCOCO/+/g, LVIS, VG referring queries and the CORE retrieval benchmark.
• Metrics: top-1 rate and association contrasts (e.g., crime−occupation, fair value 0), "the man"→Black woman (chance value 0.25), P(woman|"person") (parity 0.5), refusal AUROC, over-refusal (average refusal on should-point prompts), referring localization accuracy, and CORE P@1.
• Statistics: mosaic group as the unit, 10,000 bootstrap iterations for 95% CI; Wilcoxon tests, with all 26 headline tests surviving BH correction at q<0.05；训练模型用三种子 Welch t 检验。
• 重复与种子：CIS 与 SafeSteer 三个种子（mean±std）；参照模型在 SC 上用 300/1,050 组（Table 2 标注）。

主结果

据 Table 3（SC/FF，3 种子；公平值在括号内）：

Swipe horizontally to read the table

指标（公平值）	Released	CIS	SafeSteer
crime−occ, SC (0)	0.092	−0.002	0.014
crime−occ, FF (0)	0.047	0.002	0.016
monkey−dog, SC (0)	0.128	−0.047	0.000
"the man"→黑人女性, SC (.25)	0.756	0.302	0.294
"the person"→woman, FF (.5)	0.154	0.511	0.397
race spread, held-out (0)	0.306	0.120	0.159
"the woman" acc., FF (1.0)	0.977	0.977	0.982

• 作者称发布版存在男性默认（"the person" 仅 15.4% 选女性，ω=0 时为 54.5%）、crime 路由与对黑人女性的男性化解读；SafeSteer 去掉了大部分种族路由且不损 "the woman" 准确率。
• 作者称 CIS 在 crime 路由上去得更彻底（−0.002），SafeSteer 因拒答与不变性共享适配器而残留略多。
• 训练语料中性别中性词 63.0% 指男性，作者称模型把它放大到 84.6%；"criminal"、"thief" 从不出现在训练字幕中。

归因与范围

• 逐层与强度：置零 block 9 使 crime−occ 从 0.092 降到 0.005、"the man"→黑人女性从 0.756 降到 0.395，性别定位保持 ≥0.96（Table 8）；crime−occ 随 ω 增长：0.017→0.055→0.085→0.092（ω=0.3/0.6/0.8/1.0，Table 9）。
• 文本交换：仅给 block 9、11 提示词（其余层给 "the person"）复现 93% 的 crime 对比和 98% 的多余 "the man" 路由；反之去掉 91% 和 71%（Table 7）。
• 文本编码器无此关联：crime 词与 "Black" 的 RoBERTa 嵌入不比 occupation 词更近（d=−0.04, p=0.55）；masked-LM 头一致（0.530 vs 0.520, p=0.35），作者称关联由引导适配器零样本产生。
• 检查点扫描：两个头条种族效应在全部七个检查点上都存在，跨三种预训练目标和两种数据混合，DINOv2-B 最强（Table 2、Table 11）。

真实照片与其他模型

• PHASE/MIAP：3,096 张同时含男女的照片中 "the person" 仅 24.5% 指女性（基准率 48.8%）；MIAP 36.7%（基准率 44.4%）；ω=0 时为 58.7%、45.3%。1,026 张含黑人与他族照片中 crime 比 occupation 多指黑人 2.6 pp（p=1.9e−9）。100 张全女性含黑人女性照片中 "the man" 61.0% 选中她（基准率 42.0%）。
• 四个参照模型：均无 crime 路由（CLIPSeg −0.1 pp、G-DINO −1.0 pp、SAM 3 0.0 pp、Qwen2.5-VL +0.6 pp），"the man"→黑人女性分别为 25.3%、30.5%、42.9%（SC；真实 FF 面孔上 21.9%）、拒答（答 "there are none"）；作者称种族路由是 SteerViT 引导特有的，但 CLIPSeg、SAM 3、Qwen2.5-VL 的职业性别刻板印象更强。

拒答与修复效果

• 发布版不能拒答：热图平坦度区分应指/应拒答 AUROC 0.517（随机水平）；缺失物体上比人在场更尖锐（AUROC 0.31，低于随机）。
• 置信度不够：发布版隐藏 max patch logit 在缺失/在场达 0.880，CLIPSeg 0.883，但公平性情形（"the criminal" vs "the police officer"）CLIPSeg 0.480、logit 0.757；SAM 3 达 0.852 但以过度拒答为代价（43% 可见角色、15% benign-present 被拒），Qwen2.5-VL 拒 82% 敏感词同时拒 51% 可见角色（Table 4、Sec. 5）。
• SafeSteer：应指/应拒答 AUROC 0.997±0.001，缺失 0.995、公平性 0.998、held-out 词 0.998；0.5 阈值下拒 99.8% 训练敏感词、62% held-out 敏感词，对 benign-present 与可见角色零过度拒答（Table 16）。真实 PHASE 照片上非视觉身份提示词拒答率 49% vs 可定位提示词 1.2%（AUROC 0.935）。
• 效用保持：指代定位 0.819±0.001（发布版 0.826），CORE P@1 0.962±0.001（发布版 0.964）；等预算真值掩码微调降到 0.793，作者称是蒸馏保住了定位（Table 17）。仅拒答消融 AUROC 0.988 但偏差维持发布版水平（crime−occ 0.101），说明两个目标都需要。

其他消融与分析

• held-out 提示词：未见过的提示词上每个种族对比至少去掉 80%（crime−occ 0.135→0.026；"the boy"/"the elderly man"→黑人女性 0.605→0.267，Table 15）；例外是 held-out 中性词 "the face"，SafeSteer 的女性份额进一步低于 parity。
• 未见类别：按 16 类词汇留一训练，fold A/B 上未见类别敏感词 AUROC 0.977/0.897（发布版 logit 0.825/0.866，Table 19）。
• 迁移：同一配方迁移到 MAE-B、SigLIP-B 检查点（各分割 AUROC 均 >0.92, zero over-refusal) and CLIPSeg (sensitive/role 0.480→0.974, Table 20, 21).
• Non-prompt classifier: a text probe trained on the same words reaches only 0.748 on held-out benchmark words; a two-stage pipeline with a text gate plus logits reaches at most 0.934; SafeSteer's refusal depends on whether the object is in the image (COCO AUROC 0.89), and it has zero refusals on 12,922 real referring queries (Table 22, 23).
• Controlling for gender presentation: after controlling for CLIP masculinity score, hair length, and background brightness, the odds ratio of the Black woman being selected remains 7.6 (SC; FF 2.0); in mosaics where she is neither the most masculine nor the shortest-haired, she is still selected 72.6% of the time (Table 24).
• Other bias axes: tattooed people are selected as "the drug dealer" +0.430; negation is ignored ("ignore the woman" still selects a woman 82.4%/91.5%); the masculinization error for Black women rises with steering (0.083→0.100) and falls for other ethnicities (Appendix P).

Limitations and future directions noted by the authors

• Repair is incomplete on real cluttered scenes (Sec. 7): on fully cluttered natural scenes, the crime→Black routing is only partially reduced (2.6→1.8 pp on PHASE), and the male default can transfer; the authors suggest that closing the gap may require box-level supervision on natural images such as FACET.
• Sharing an adapter between refusal and invariance brings a trade-off (Sec. 7): SafeSteer retains more crime routing than the debiasing-only CIS; and it never refuses on nationalities and immigration-related words unseen in real photos.
• Side effects of CIS (Sec. 7): the debiasing-only CIS weakens localization of explicit race words ("the Black person"), which the authors describe as a side effect expected under invariance.
• Release plan (abstract): the authors plan to release code, the benchmark, and the curated mosaic set.

Scope of experiments

• The models tested are seven released SteerViT checkpoints plus four reference localization models (CLIPSeg, Grounding-DINO, SAM 3, Qwen2.5-VL-7B-Instruct), 11 in total; SafeSteer is additionally validated on MAE-B and SigLIP-B checkpoints and on CLIPSeg (Sec. L, M).
• Evaluation data cover three mosaic sources (FairFace, SocialCounterfactuals, PAIRS) and two real-photo sets (PHASE 4,317 images, MIAP 2,886 images); utility covers 12,922 RefCOCO-family queries and CORE retrieval (Appendix A, I).
• Race and gender labels come from dataset annotators labeling by appearance or from generation prompt settings, not self-identification; binary gender categories follow the datasets (Appendix A).
• Training-based results (CIS, SafeSteer) use three seeds; SafeSteer on other backbones (MAE-B, SigLIP-B) uses a single seed, and the unseen-category fold B uses a single seed (Table 19, 20).
• The paper does not report the compute cost or time of SafeSteer training; reference models on SC use only 300/1,050 groups (as noted in Table 2).

A paper auditing and repairing the fairness of a text-steerable vision encoder. The authors ask: when a prompt's referent cannot be judged from the image, does SteerViT decide whom to point to based on perceived race/gender? Can it refuse?

• Audit method: counterfactual mosaics (four people differing in only one protected attribute, four rotations to eliminate position priors) plus real multi-person photos (PHASE, MIAP), with attribution via layer-wise gate zeroing, text interchange intervention, and frozen text encoder checks.
• Main findings (released version): "the person" selects male 84.6% of the time; on SC, crime points to Black people 0.092 more than occupation; "the man" among four women selects a Black woman 75.6% of the time; "the monkey" points to Black people 12.8 pp more than "the dog". The authors state these grow with steering strength, enter through the last two cross-attention layers, the text encoder does not contain this association, and all seven checkpoints exhibit them; four reference localization models have no crime routing.
• Inability to refuse: the patch softmax must normalize, and the released version's AUROC for distinguishing should-point/should-refuse is only 0.517; the confidence signal handles object absence (about 0.88) but does not exceed 0.757 in fairness cases, and SAM 3 and Qwen2.5-VL refuse sensitive words by over-refusing visible roles.
• SafeSteer: adds a learnable null logit after the N patch logits for refusal output, with counterfactual invariance training (same 23M steering parameters, 4k steps). Should-point/should-refuse AUROC 0.997, held-out words 0.998, zero over-refusal on the benchmark; crime−occ drops to 0.014, "the man"→Black woman drops to 0.294; localization 0.819 (released version 0.826), CORE P@1 0.962 (0.964).
• Authors' conclusions: the authors state that language steering injects social stereotypes into where the model points, and that refusing stereotype-inducing queries should be a first-class property of the steering interface; limitations are that crime routing is only partially eliminated on real scenes, there is a trade-off with refusal, and CIS weakens localization of explicit race words.