Stealing Patch Size: Adversarial Manipulation of Vision-Language Models

Kai Hu and other researchers propose a black-box model-stealing attack that can recover private visual tokenizer configurations from deployed vision-language models (VLMs), including visual patch size and the input preprocessing pipeline. Its principle is a task-level side channel introduced by ViT-style image patching: when a synthetic grid image aligns with the hidden patch grid, boundary cues at the tokenization stage are erased, causing periodic accuracy drops; scanning grid cell sizes and measuring these accuracy drops allows inferring the patch size, and then introducing padding and consistency checks distinguishes dynamic or fixed-resolution preprocessing and recovers the target scaling resolution. The attack can reliably recover tokenizer-related parameters on open-source Qwen-VL variants as well as proprietary models such as GPT and Claude. The researchers also show that such leakage can support transfer attacks that perceive preprocessing and model-specific adversarial manipulation.