SAFIRE: A safety-critical benchmark for fine-grained flame and smoke understanding in multimodal large language models

The researchers propose SAFIRE, a large-scale benchmark for flame and smoke understanding aimed at multimodal large language models (MLLMs). Evaluation of 10 open-source MLLMs (8B-38B) yields an average accuracy of 61.9%, exposing clear gaps in safety-critical reasoning. The benchmark contains 83K annotated images from 20 scenarios and generates 193K multiple-choice visual question answering (MCVQA) items from a subset of 9.7K images, covering 10 evaluation dimensions from basic perception to higher-order reasoning; annotation quality is ensured by a GPT-5.4-assisted multi-stage verification pipeline combined with MLLM majority voting. Experiments also show that adapting the vision encoder with only 7% domain-specific data raises fire-scene classification accuracy from 20.1% to 64.5%. The dataset, models, and code are released at https://risys-lab.github.io/SAFIRE/。