Shift-Reliance Alignment: ranking vision-language models by the cues they trust

Researchers propose the Shift-Reliance Alignment (SRA) method, which measures vision-language models' reliance on cues such as scene context and text in images to predict which model will have the lowest error without trial-running candidate models in the target domain; calibrating the detector requires 650 to 1,200 labeled target samples. The method edits 100 labeled source images so that cues point to wrong answers to quantify reliance; on nine models not involved in designing the metric, the degree of reliance predicted misdirection by physical sticker attacks with a Spearman of 0.93, and also predicted model hallucination on unedited COCO photos. In a preregistered test on new images where misleading questions were the majority, the model selected by SRA reduced error from 30.4% to 18.8% on Objects365 and OpenImages compared with model selection by source-domain error, but did not work as predicted on COCO. The researchers also release the CueDial benchmark; when overlaid text denies visible objects, selection by model confidence trails the best model by more than 40 percentage points, while SRA stays within 3 percentage points.