VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Model Alignment

The paper proposes the VLFeedback dataset, containing over 82K multimodal instructions and feedback rationales generated by off-the-shelf models without human annotation; the authors call it the first large-scale vision-language feedback dataset. Using this dataset, the research team trained the LVLM Silkie via direct preference optimization (DPO); the paper reports that it surpasses the base model by 6.9% and 9.5% on perception and cognition tasks, respectively, reduces hallucination problems on MMHal-Bench, and shows stronger resistance to red-team attacks, with improvements in helpfulness, visual faithfulness, and safety metrics. The analysis notes that the advantage of AI feedback lies in promoting preference diversity, thereby bringing more comprehensive improvements. The dataset, training code, and model have been open-sourced at https://vlf-silkie.github.io.