Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

Vision-language models need to correct, abstain, or refuse only the incorrect, invisible, unknowable, infeasible, or unsafe components within the same query, while continuing to answer the remaining answerable components.

• Scenario: The authors state that VLMs are increasingly used in real-world applications, where helpful and contextually appropriate responses are a core design goal; when faced with false assumptions, information not inferable from the image, or unsafe or infeasible requests, they should correct premises, express uncertainty, or refuse the relevant components rather than give misleading answers.
• Existing Gaps: The authors state that existing benchmarks primarily evaluate non-compliance on entire queries, assuming a request is either fully compliant or fully non-compliant. Real queries can mix answerable content with components that should be non-compliant, and selective handling still lacks dedicated evaluation.
• Paired Observation: The same image and the same source of non-compliance can be paired into single and compound queries. The authors state in the introduction that compound queries can include one or more answerable components, to test whether the model refuses, corrects, or abstains only on the affected component.
• Paper Contribution: The authors propose the benchmark KoNA, covering False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety, and evaluate both query-level and component-level non-compliance. The authors also fine-tune VLMs using selective non-compliance samples and fully answerable samples.

Visual Question Answering and Answerability Assumptions

• Standard VQA: Antol et al. (2015), Goyal et al. (2017), etc., evaluate factual answering and visual grounding on images and language. The authors state that such settings typically assume questions are valid and answerable.
• Priors, Hallucination, and Isolated Adversarial Settings: The authors state that models may rely on language priors or hallucinate visual details (Agrawal et al., 2016; Clark et al., 2019; Whitehead et al., 2022). Adversarial or counterfactual questions are mostly evaluated in isolated settings (Liang et al., 2020; Li et al., 2021; Wu et al., 2024a).

Language-Side Non-Compliance

• Refusal in Alignment: Amodei et al. (2016), Askell et al. (2021), Bai et al. (2022a) discuss that models should be helpful while being non-compliant in inappropriate situations.
• Specific Mechanisms: Policy-driven refusal (Ouyang et al., 2022; Bai et al., 2022b), false premise detection (Ray et al., 2016), and abstention or uncertainty calibration when unanswerable (Rajpurkar et al., 2018; Brahman et al., 2024).

Multimodal Non-Compliance

• Hallucination and False Premises: Rohrbach et al. (2018), Li et al. (2023b), Wang et al. (2024), Wu et al. (2024b), etc., address visual hallucination or false premises in visual tasks.
• Safety and Answerability: Qi et al. (2024), Liu et al. (2024b), Li et al. (2024) address visual safety; Gurari et al. (2018), Guo et al. (2024), Eisenschlos et al. (2024), Miyai et al. (2025), Zhu et al. (2026) address answerability and abstention. The authors state that this line of work typically evaluates at the query level and expects the entire response to be either compliant or non-compliant.

Baseline Methods and Benchmarks

• Baselines: Default inference, Chain-of-Thought (Wei et al., 2022), Behavior Guidance prompting; training comparisons include SFT with and without the answerable set, full SFT, and SFT plus GRPO.
• Benchmarks: Main evaluation uses KoNA. Extensions use HaloQuest, MM-SafetyBench, R-Bench, UPD; general capabilities use TextVQA, MIA-Bench, MMBench, POPE; over-refusal uses MOSSBench.

The authors position this paper in the setting where non-compliance conditions are embedded in broader queries: evaluating whether the model can answer the valid components while refusing, correcting, or abstaining on the remaining components as appropriate.

The authors use the benchmark KoNA to make five types of non-compliance comparable across single and compound queries, then fine-tune two open-source VLMs using supervised fine-tuning (SFT) plus Group Relative Policy Optimization (GRPO) on mixed samples. The categories are adapted from prior work on non-compliance and VQA reliability, keeping single and compound queries comparable.

Five Types of Non-Compliance

• False Premise and Inaccessibility: False Premise targets verifiable visual attributes, changing only one precise detail while the rest of the description matches the image; the expectation is to correct using visual evidence, not to accept silently. Visual Inaccessibility asks about attributes present in the scene but not determinable due to occlusion, blur, viewpoint, or lighting; the expectation is to state that it is not visible, not to infer or hallucinate.
• Unverifiable Assumptions: Universal Unknown involves assumptions about relationships, roles, or intentions that cannot be verified from the image alone. The expectation is to clearly state that it cannot be verified; only qualified inferences are acceptable if clearly presented as possible rather than confirmed.
• Infeasibility and Safety: Task Feasibility requests actions beyond the capabilities of the evaluated VLM, such as physical tasks; the expectation is to acknowledge that the infeasible component cannot be performed while answering the feasible components. This criterion is limited to VLMs and not used for models with other capabilities like image generation. Safety requests unethical, malicious, or unauthorized actions involving clearly visible objects or places in the image; the expectation is to recognize the harmful intent and explicitly refuse to assist.

Three-Stage Data and Filtering

• Single Queries: Images are randomly sampled from MS COCO and Open Images V7. Each single query targets only one non-compliance condition from Section 3.1, used to measure query-level non-compliance.
• Compound and Fully Answerable: Section 3.2 adds an additional, image-answerable component to each single query. Finally, the non-compliant component is changed to a fully answerable form, preserving the original query structure; the corresponding answer is a direct, image-based response without correction, refusal, or uncertainty.
• Generation and Filtering: Query-answer pairs are generated by GPT-5 and Gemini-2.5-Flash under fixed configurations. Automatic filtering checks consistency, task definition, visual grounding, and query-answer correspondence. The test set is fully verified via Amazon Mechanical Turk; extractions with token overlap below 0.2 are discarded, and 10% fully answerable queries are inserted to check workers. Only samples passing both automatic filtering and human verification are retained.
• Scale: The final set includes 3,100 image-level instances, each with single QA, compound QA, and fully answerable QA, totaling 9,300 QA pairs (Table 1). The five categories are balanced. Training, validation, and test splits are 1,300, 300, and 1,500, respectively.

SFT and GRPO

• Two Stages: SFT primarily learns component-level non-compliance on compound queries while handling answerable components; then GRPO balances non-compliance with factual accuracy on answerable components. Fine-tuning targets InternVL3-2B-Instruct and Qwen2.5-VL-3B-Instruct, with vision encoders frozen.
• Data Allocation: Of the 1,300 training instances, 1,200 are used for SFT and 100 for GRPO. SFT includes 1,000 compound, 100 answerable, and 100 single queries. GRPO randomly samples 80 compound and 20 answerable, balanced across tasks and image sources, and non-overlapping with the SFT set.
• Rewards: GPT-5-mini provides Rnon (failure if component-level non-compliance is not passed) and Rfac (whether the answerable response is image-based). If Rnon fails, the reward is 0; if passed, it is 1.0−λ· 1(Rfac=FAIL). Experiments use λ=0.3. The authors state this structure prioritizes selective non-compliance while penalizing factual errors on valid components.
• Training Amount: SFT runs for 3 epochs; GRPO for 30 epochs, sampling 4 responses per input, with KL regularization relative to the reference policy. Learning rate and other details are in Appendix B.1.

Success Criteria and Inference Settings

• Three Proportions: All computed by GPT-5-mini as PASS. Query-level checks whether the model identifies the trigger for that category and provides the expected correction, abstention, or refusal. Component-level applies to compound queries, requiring selective handling of the invalid component and accurate handling of the answerable component. Factual accuracy checks whether the answerable component is factually correct and visually grounded.
• Fully Answerable Control: The component-level review prompt in the appendix includes a Contrast criterion: fully answerable queries should be answered directly, without unfounded refusal, correction, or uncertainty.
• Inference Variants: Beyond default inference, Chain-of-Thought appends "Let's think step by step." after the query; Behavior Guidance explicitly instructs when to correct premises, express uncertainty, or refuse actions. Open-source models use vLLM greedy decoding with a maximum of 2,048 tokens; closed-source models use the default settings of public APIs.

Experimental Setup

• Models tested: InternVL3-2B-Instruct, InternVL3-78B-Instruct, Qwen2.5-VL-3B-Instruct, Qwen2.5-VL-72B-Instruct, GPT-5, Gemini-2.5-Flash, and fine-tuned InternVL3-2B-KoNA, Qwen2.5-VL-3B-KoNA.
• KoNA test: 1,500 image-level instances (Table 1), equally distributed across five categories; images from MS COCO and Open Images V7, generators are GPT-5 and Gemini-2.5-Flash. Each instance has a single query, a compound query, and a fully answerable query.
• Controls: default inference, Chain-of-Thought, Behavior Guidance; appendix also has 5-shot. Training controls include SFT without the answerable set, SFT with that set, full SFT, and SFT+GRPO.
• Metrics: Table 2 shows single/compound results for five task accuracies and two averages. Table 3 reports Answerable Average and Factuality Average under default inference. Section 4.3 additionally defines query-level, component-level non-compliance accuracy and factual accuracy.
• Evaluation: Main metrics judged by GPT-5-mini. On 640 outputs, agreement with human is query-level 94.0%, component-level 93.8%, factuality 97.0%, overall 94.8%; Gemini-2.5-Flash re-evaluation agrees with GPT-5-mini overall 95.2% (Appendix C). On the GRPO training set, GPT-5-mini's reward agreement with Gemini-2.5-Flash and Claude Haiku 4.5 is 85.2% and 90.1%, with Pearson correlation to the mean of two alternative judges of 0.887 (Appendix B.1).
• Decoding: Open-source models use greedy decoding. The paper does not report the number of repetitions for main results.

Main Results

Model	Single Average	Compound Average
InternVL3-2B	0.25	0.10
InternVL3-78B	0.35	0.23
Qwen2.5-VL-3B	0.29	0.11
Qwen2.5-VL-72B	0.59	0.34
GPT-5	0.55	0.42
Gemini-2.5-Flash	0.49	0.39
InternVL3-2B-KoNA	0.92	0.90
Qwen2.5-VL-3B-KoNA	0.91	0.87

From Table 2. The first six rows are Default Inference, the last two are Fine-Tuned with KoNA.

• Compound queries: The authors state that for most models on most tasks, compound query accuracy is lower than single query, indicating difficulty in separating components that should not be followed from answerable components. At the model average level, open-source models often show larger single-compound gaps than closed-source, but not uniformly. There are also ties under default settings, such as InternVL3-78B's Task Feasibility at 0.04/0.04 and Gemini-2.5-Flash's Task Feasibility at 0.12/0.12.
• Category differences: Under default settings, Task Feasibility is low, with Qwen2.5-VL-72B at 0.14/0.09. GPT-5's Safety is 0.95/0.91, but False Premise compound is only 0.14; Qwen2.5-VL-3B's Safety is 0.43/0.01 (Table 2).
• Fine-tuning and answerability: The authors state that fine-tuning reduces the gap between single and compound queries, and the improvement is not limited to a single category. Under default settings in Table 3, the two fine-tuned models have Answerable Average of 0.70 and 0.71, with corresponding base models at 0.77 and 0.73; Factuality Average is 0.88 and 0.89, with corresponding base models both at 0.84. The authors state that fine-tuned models largely maintain accuracy on fully answerable queries.

Inference-time Prompts

• Chain-of-Thought: Changes are unstable and have limited impact on compound queries. Qwen2.5-VL-3B's Single/Compound Average goes from 0.29/0.11 to 0.18/0.10; Qwen2.5-VL-72B from 0.59/0.34 to 0.43/0.32 (Table 2).
• Behavior Guidance: Improvements are larger for some larger models, and for Task Feasibility and Safety. Qwen2.5-VL-72B is 0.87/0.79, Gemini-2.5-Flash is 0.85/0.77; InternVL3-2B's Compound Average is 0.08, lower than its default 0.10. The authors state that the prompt itself cannot stably support selective non-compliance in compound queries; improvements on False Premise and Universal Unknown are more limited because they require more nuanced, context-dependent responses rather than direct refusal.

Cross-benchmark, General Capabilities, and Failure Modes

• Extended benchmarks: Original queries as single queries, then add answerable components following the same procedure. UPD uses only IVQD, excluding AAD and IASD because the latter two's unsolvability mainly comes from missing or incompatible options. Base models often have lower compound queries, such as InternVL3-78B's MM-SafetyBench at 0.88/0.05 (Table 5). After fine-tuning, InternVL3-2B-KoNA's four compound scores are 0.62, 0.60, 0.56, 0.60. Exceptions: Qwen2.5-VL-72B's HaloQuest is 0.67/0.79; Qwen2.5-VL-3B-KoNA's MM-SafetyBench single query is 0.65, lower than base 0.75.
• General capabilities and refusal rates: In Table 6, InternVL3-2B after fine-tuning still has TextVQA at 0.79, MIA-Bench from 0.64 to 0.62, MMBench from 0.80 to 0.76, POPE from 0.92 to 0.93, MOSSBench refusal rate from 0.01 to 0.04. Qwen2.5-VL-3B correspondingly from 0.83 to 0.84, 0.70 to 0.69, 0.79 to 0.76, 0.89 to 0.90, 0.02 to 0.05. The authors state no broad decline in these capabilities, and refusal rates on safety queries remain low. WildGuard re-evaluation is 0.03 to 0.04, 0.03 to 0.06 (Table 7).
• Error modes: Section 6.4 and Figure 3 give three types: on Task Feasibility, refusal extends to instructions that could be textually answered; on Visual Inaccessibility, abstains on unreadable details but misses answerable attributes; on False Premise, recognizes the false premise but the correction itself is inaccurate. The authors state that selective non-compliance also requires preserving all answerable content and basing corrections on accurate visual evidence.

Other Ablations and Analyses

• Table 4: Without the answerable set, InternVL3-2B-KoNA and Qwen2.5-VL-3B-KoNA have Answerable Average of 0.38 and 0.53; with it, 0.60 and 0.64; adding GRPO gives 0.70 and 0.71. The authors state that without that set, unnecessary refusals occur.
• Table 10: Full SFT without GRPO has Answerable Average of 0.62 and 0.65, lower than SFT+GRPO's 0.70 and 0.71. The authors believe GRPO helps achieve a better balance between non-compliant queries and fully answerable queries.
• Table 11: With GRPO samples 50/100/150, InternVL's Compound Average is 0.84/0.90/0.90, Answerable Average is 0.59/0.70/0.71. The authors state that increasing to 150 does not provide consistent additional gains.
• Table 8: CC3M out-of-distribution test with 750 items, 150 per category. Fine-tuned models have Compound Average of 0.88 and 0.86, with corresponding 2B/3B base models at 0.11 and 0.07.
• Table 9: In 5-shot compound average, InternVL3-78B is 0.64, Qwen2.5-VL-72B is 0.68, Gemini-2.5-Flash is 0.71; the two fine-tuned models remain at 0.90 and 0.87. The authors state that larger VLMs can better utilize in-context examples.
• Appendix K: On a 250-question VizWiz-VQA subset, selective non-compliance rate goes from InternVL3-2B's 0.32 to 0.72, Qwen2.5-VL-3B's 0.39 to 0.73. The authors state that this subset is filtered by compound structure and partial unanswerability, the original question text is not rewritten, and it should not be interpreted as performance on the full VizWiz-VQA distribution.

Limitations and Future Directions Noted by the Authors

• Model scale: The experiments do not include open-source VLMs with more than 80B parameters; the authors state the reason is computational constraints (Limitations).
• Modality scope: The research targets only VLMs and does not consider other modalities such as audio models or image generation models (Limitations).
• Future directions: The authors state that the task formulation and evaluation framework can be extended to more modalities and model categories, and explore a more complete benchmark integrating non-compliance evaluation across tasks, modalities, and model scales (Limitations).

Experimental Coverage

• Main evaluation models are InternVL3-2B/78B-Instruct, Qwen2.5-VL-3B/72B-Instruct, GPT-5, Gemini-2.5-Flash; parameter updates were done only on InternVL3-2B-Instruct and Qwen2.5-VL-3B-Instruct (Section 4, Table 2).
• KoNA test split has 1,500 image-level instances, evenly distributed across five categories; images come from MS COCO and Open Images V7, queries are generated by GPT-5 and Gemini-2.5-Flash, and test instances underwent automatic filtering and MTurk verification (Table 1, Section 3).
• Judgment is done by GPT-5-mini; human agreement rate is based on 640 outputs (Appendix C). Open-source inference uses greedy decoding. The paper does not report the number of repetitions for the main results.
• Additional evaluations include composite query extensions of HaloQuest, MM-SafetyBench, R-Bench, UPD, as well as TextVQA, MIA-Bench, MMBench, POPE, MOSSBench (Table 5, Table 6).
• Appendices also report 750 out-of-distribution tests on CC3M, 5-shot in-context learning, GRPO sample counts of 50/100/150, and a 250-question VizWiz-VQA subset (Appendices H, I, J, K).

KoNA 用来评测视觉语言模型能否对混合查询做选择性不遵从：只处理不该遵从的成分，同时回答其余成分。

• 问题：作者称现有评测多把不遵从当成整条查询的二元选择。一条查询却可以同时含可答成分，以及应纠正、弃权或拒绝的成分。
• 做法：五类任务各有成对的单查询和复合查询，并另建全可答对照。图像来自 MS COCO 与 Open Images V7，共 3,100 个图像级实例、9,300 个 QA 对。作者再用 SFT 和 GRPO 微调 InternVL3-2B-Instruct 与 Qwen2.5-VL-3B-Instruct；成分级不遵从失败时奖励为 0。
• 默认结果：Table 2 中，InternVL3-2B 的单/复合平均为 0.25/0.10，Qwen2.5-VL-72B 为 0.59/0.34，GPT-5 为 0.55/0.42。多数模型的复合查询更低，但差距并不整齐。
• 微调：InternVL3-2B-KoNA 为 0.92/0.90，Qwen2.5-VL-3B-KoNA 为 0.91/0.87。全可答平均为 0.70 与 0.71（Table 3）。去掉 answerable 训练集后，这两项可答平均为 0.38 与 0.53（Table 4）。
• 提示并不替代训练：Behavior Guidance 下 Qwen2.5-VL-72B 复合平均为 0.79，InternVL3-2B 仅为 0.08（Table 2）。作者认为微调后的模型能够区分可答成分与需要不遵从的成分；第 6.4 节同时给出过度拒绝、漏答和纠正错误。作者在结论中称，这项工作指出了现有 VLM 的关键限制，并为发展更可靠的不遵从行为提供基础。