Fabricated Answers: Adversarial Fabrication Attacks against OCR-Free Document Visual Question Answering

The researchers propose an adversarial fabrication attack against OCR-free document visual question answering (DocVQA) that can make visually imperceptible but semantically targeted alterations to document content, inducing the model to give specific or generally incorrect answers. The method designs dedicated algorithms for different attack goals, covering scenarios ranging from targeted disinformation to systematic model failure, and demonstrates effectiveness on two end-to-end models, Pix2Struct and Donut. The authors state that this exposes critical vulnerabilities in current DocVQA systems, call for the development of more robust defenses, and have open-sourced the code (https://github.com/pralab/adv-docVQA）。