ChartHal: A Fine-Grained Benchmark for Evaluating Hallucination in Chart Understanding by Large Vision-Language Models

Hallucination in chart QA still lacks evaluation broken down by triggering scenario.

• Scenario: The authors believe charts are information-dense with complex element relationships, so models need both precise perception and cognitive reasoning, and must strictly align with the visual input. The authors define hallucination as generating information that does not exist in reality or is inconsistent with the input, which is especially problematic in chart tasks.
• Existing gaps: The authors state that previous hallucination research and chart understanding have been conducted separately, with almost no exploration of their intersection. Hallucination evaluation is mostly limited to objects and scenes; chart benchmarks mostly report answer accuracy, rarely explaining whether hallucination occurs and how.
• Author's judgment: Questions asking for information not in the chart, or where the question premise contradicts the chart, are more likely to induce hallucination. The authors' reasoning is that LVLMs tend to generate answers regardless of whether the question can be answered from the chart.
• This paper proposes: The authors propose the benchmark CHART HAL, which uses chart QA to categorize chart-question pairs into 12 hallucination triggering scenarios, and provides 1,062 human-verified samples. The introduction states that to the best of their knowledge this is the first fine-grained benchmark for this task; the conclusion also calls it the first systematic benchmark.

The authors split related work into two lines, hallucination evaluation and chart understanding, and state that their intersection is almost unexplored.

Hallucination definition and existing evaluations

• Definition: The authors cite Huang et al. (2025), describing hallucination as information that is absent from or factually incorrect in the input; on the LVLM side, they emphasize inconsistency with the visual input (Bai et al., 2024). The authors state the phenomenon has been widely observed since the advent of LLMs, citing Hurst et al. (2024), OpenAI (2025a,b), Comanici et al. (2025).
• Objects and attributes: Rohrbach et al. (2018), Li et al. (2023), Zhang et al. (2023), Fu et al. (2023), Zhang et al. (2024), Sun et al. (2024), Qiu et al. (2024). The paper groups these as work focusing on objects and attributes in input images.
• Scene-level: Zhou et al. (2023), Wang et al. (2023), Liu et al. (2023a), Jiang et al. (2024). The authors state that overall, these works leave a major gap in chart hallucination evaluation.

Hallucination benchmarks involving charts

• HallusionBench (Guan et al., 2023): The authors state this is, to their knowledge, the only benchmark involving LVLM chart hallucination, but questions are limited to Yes/No, simplifying the chart understanding setting.

Chart understanding benchmarks

• Accuracy-oriented: FigureQA (Kahou et al., 2017), DVQA (Kafle et al., 2018), PlotQA (Methani et al., 2020), ChartQA (Masry et al., 2022), ChartBench (Xu et al., 2023), MMC (Liu et al., 2024), CharXiv (Wang et al., 2024c), ChartX and ChartVLM (Xia et al., 2024), ChartQAPro (Masry et al., 2025), ChartMuseum (Tang et al., 2025), InfoChartQA (Lin et al., 2025). The authors state that most of these look at whether answers are correct rather than analyzing hallucination.
• Unanswerable subsets: CharXiv and ChartQAPro. The authors state this enables limited hallucination evaluation, but the subsets remain small and lack fine-grained classification by hallucination-triggering question type.
• ChartCap (Lim et al., 2025): The authors state it studies hallucination in chart understanding, but is limited to captioning and does not cover interactive question answering.

Baselines and data

• Comparison targets: The paper does not treat other systems as method baselines to surpass. Experiments compare scores of 15 LVLMs on CHART HAL; charts are sampled from CharXiv. Whether hallucination occurs is judged by GPT-4o according to the criteria in Section 3.3.

The authors position this paper in the gap above: using free-form answers and fine-grained classification by question type and chart-question relationship, rather than only reporting final accuracy on chart question answering.

CHART HAL turns evaluation into free-form chart question answering, then classifies each question into 12 categories by question type and chart-question relationship.

Task format

• Input-output: Given chart image c and question q, the LVLM generates answer a. The experiment provides only these two items, without instructions like "output a fixed phrase if unanswerable"; the authors state this differs from CharXiv (Appendix B).
• Why free-form: The authors do not use Yes/No or multiple choice, claiming free-form better reflects hallucination behavior in real chart understanding, citing Li et al. (2024).
• Two dimensions: Figure 1 upper half shows the construction pipeline, lower half illustrates categories with examples. The paper states the two dimensions are independent.

12 category scenarios

• Question types: descriptive, directly extracting information from the chart; reasoning, performing calculations or logical inference based on the chart; open-ended, analysis or prediction without a definite correct answer.
• Chart-question relationships: irrelevant, unrelated to the chart; inexistent, asking for information missing from the chart; contradictory, premises inconsistent with chart data; normal, valid questions consistent with the chart. The first three are deemed unanswerable.
• Usage assumption: The authors believe unanswerable questions are more likely to induce hallucination because models tend to answer regardless of answerability (Kalai et al., 2025).
• Sample sizes: In Table 1, irrelevant 269, inexistent 344, contradictory 210, normal 239; descriptive 383, reasoning 322, open-ended 357; total 1,062. The smallest cross-cell is reasoning×contradictory with 54 items.

Data construction

1. Chart sampling: Stratified sampling from the CharXiv validation set (Appendix A.1). Original sub-chart counts range from 1 to 120; only 1 or 2 sub-charts are kept, in a 3:2 ratio. Then sampled by discipline proportion, covering bar charts, line charts, etc. The paper does not report the final number of charts.
2. QA generation: o4-mini generates questions and reference answers covering the 12 subcategories for each selected chart. Appendix Tables 3–14 provide generation prompts for each category.
3. Human verification: The main text says expert annotators; the appendix specifies undergraduate annotators. Pairs clearly inconsistent with the design are discarded against the chart, factual errors are corrected, categories are relabeled, and standard answers are modified (acceptable ranges given when readings are approximate). The paper does not report the number of annotators, discarded items, or agreement.

How no-hallucination is determined

• Scoring: No hallucination scores 1, otherwise 0. Table 2 normalizes aggregate scores to [0,100]. The abstract calls these scores accuracy.
• General rule: Consistent with the given reference answer, or explicitly stating the question is unanswerable.
• Category details (Appendix A.2): Irrelevant questions must point out irrelevance and not fabricate answers. Inexistent descriptive and reasoning questions must state missing information and not guess. Open-ended inexistent questions may declare unanswerability or speculate with explicit uncertainty. Contradictory questions must point out the contradiction, then declare unanswerability or answer after ignoring the erroneous premise; for descriptive and reasoning questions, if a numeric acceptable range is given, falling within it counts as correct. Open-ended contradictory answers must remain uncertain; being overly confident is marked wrong. For normal questions, numbers without a range must be numerically equal, allowing different notations; categorical answers must point to the same item; open-ended normal questions are judged on uncertainty expression and reasonable reasoning, not factual correctness; others are judged on semantic alignment with the reference answer, without omission or added irrelevant information.
• Review prompt: The experiment uses GPT-4o to apply the above criteria. Appendix Tables 15–21 give scoring prompts. Table 17's description allows open-ended inexistent questions to speculate with uncertain tone, while the scoring sentence also says to avoid guessing and mark fabrication as 0.

The authors report CHART HAL category scores on 15 LVLMs and state that unanswerable scenarios are more likely to induce hallucination.

Experimental setup

• Tested models: 15. Closed-source: Gemini-2.5-Pro, GPT-5, GPT-5-mini, GPT-5-nano, o4-mini, GPT-4o-2024-11-20. Open-source: Llama 3.2-11B-Vision-Instruct, InternVL2.5-4B/8B/38B/78B, Qwen2.5-VL-3B/7B/32B/72B-Instruct. The high row in Table 2 sets reasoning effort to high; the paper still counts 15 models.
• Data: All 1,062 CHART HAL items, distribution in Table 1. Charts sampled from CharXiv. QA generated by o4-mini, which is also among the tested models.
• Input and decoding: Only chart and question given. Temperature 0; Appendix B states reasoning models like o4-mini cannot set temperature. Generation length uses each model's default limit. The paper does not report repetition counts.
• Metric: Each item scores 1 if no hallucination, 0 otherwise, normalized to [0,100]. It is not a single factual accuracy: open-ended normal questions consider uncertain tone and reasonable reasoning (Appendix A.2).
• Review: GPT-4o-2024-11-20, temperature 0. It is also a tested row in Table 2. The appendix says using GPT-4o for scoring is common in other benchmarks, aligns with human judgment, and is more efficient and scalable; the paper does not provide human agreement numbers on this benchmark.
• Comparison: No separate method baseline; models are compared on the same set of questions.

Main results

Swipe horizontally to read the table

Model	Overall	Irrel.	Inexist.	Contra.	Normal
Qwen2.5-VL-72B-Instruct	54.24	65.06	69.77	19.05	50.63
Gemini-2.5-Pro	49.34	36.80	52.62	49.52	58.58
GPT-5 (high)	34.46	20.82	29.07	19.52	70.71
GPT-4o-2024-11-20	32.49	27.88	36.92	6.67	53.97
Qwen2.5-VL-7B-Instruct	25.71	25.65	25.29	1.90	47.28
Llama-3.2-11B-Vision-Instruct	24.95	29.37	28.49	0.00	36.82
o4-mini (high)	22.79	10.78	14.53	3.33	65.27
Qwen2.5-VL-3B-Instruct	8.57	0.74	0.87	0.00	35.98

According to Table 2, values are scores normalized to [0,100]. Omitted rows include GPT-5 (34.37), GPT-5-mini (high, 26.74), GPT-5-mini (27.02), GPT-5-nano (high, 24.95), GPT-5-nano (23.07), o4-mini (22.41), InternVL2.5-78B (21.47), InternVL2.5-38B (19.40), InternVL2.5-4B (12.71), InternVL2.5-8B (12.34), Qwen2.5-VL-32B-Instruct (23.92). InternVL2.5's four sizes also have Contra. 0.00.

• Overall score: The authors mark Qwen2.5-VL-72B as highest overall, claiming about 5 points above Gemini-2.5-Pro. In the same table, Gemini has higher Contra. and Normal, while Qwen has higher Irrel. and Inexist. The authors say o4-mini is unexpectedly poor, behind GPT-4o, and comparable only to smaller models like Qwen2.5-VL-7B.
• Unanswerable: The authors report o4-mini correctly identifies only 14.53% of inexistent questions as unanswerable, dropping to 3.33% for contradictory questions, contrasted with Qwen2.5-VL-72B's 69.77%. These numbers match the o4-mini (high) row in Table 2; the text does not specify whether it refers only to high. The authors think this resembles a bias toward generating answers rather than judging answerability. The Inexist. column per appendix rules is not entirely "acknowledging unanswerable": open-ended inexistent questions may speculate with uncertain tone.
• Abstract: The abstract states GPT-5 and o4-mini overall accuracy is only 34.46% and 22.79%. These two numbers match the GPT-5 (high) and o4-mini (high) rows in Table 2, respectively.

Question types and chart-question relationships

• Question types: The authors claim difficulty hierarchy is highest for descriptive (top models above 60%), reasoning in the middle, open-ended most error-prone, with the strongest models struggling to exceed 45%. In Table 2, descriptive above 60 is Gemini-2.5-Pro's 60.31; Qwen2.5-VL-72B descriptive is 58.49. Highest open-ended is that 72B model's 44.82.
• Cells inconsistent with hierarchy: Qwen2.5-VL-72B's Reason is 59.63, higher than its Desc. 58.49. InternVL2.5-8B's Open is 10.64, higher than its Reason 8.39. Gemini-2.5-Pro has Desc. 60.31, Reason 50.31, Open 36.69, consistent with the authors' described order.
• Relationship: The authors say models perform reasonably on normal questions, but hallucinate heavily when answerability is involved; contradictory questions are especially hard, with many open-source models at 0%. Contradictory questions total 210 (Table 1). Contra. 0.00 includes Llama-3.2-11B-Vision-Instruct, InternVL2.5's four sizes, Qwen2.5-VL-3B-Instruct. The authors believe standard questions are manageable, but unanswerable scenarios still cause hallucination, and point to the need for strategies like adversarial training.

Scale, reasoning intensity, and examples

• Scale: The authors say larger models are generally better (Qwen2.5-VL and GPT-5 series), and believe scale can enhance robustness to hallucination. The same table is not everywhere larger-is-higher: Qwen2.5-VL-32B overall 23.92, 7B 25.71; InternVL2.5-8B 12.34, 4B 12.71. Section 5 says larger models are usually better, but smaller models can occasionally exceed larger ones in specific subcategories. In Table 2, GPT-5-mini's Normal is 79.92, higher than GPT-5 (high)'s Normal column.
• Reasoning intensity: The authors say increasing reasoning effort yields only small and unstable gains, sometimes worse; the reason is that correct reasoning depends on accurate visual perception, and extra reasoning often amplifies errors when the chart is misread. Table 2 overall scores: GPT-5 (high) 34.46, GPT-5 34.37; GPT-5-mini (high) 26.74, GPT-5-mini 27.02; o4-mini (high) 22.79, o4-mini 22.41.
• Examples: Figure 2 gives three columns; caption says green text is correct parts, red text is wrong or hallucinated parts. Columns are labeled descriptive+inexistent, open+contradictory, reasoning+normal. The authors say: increasing reasoning intensity yields little benefit (GPT-5 in left and middle columns); GPT-5 series hallucinate easily in unanswerable scenarios; GPT-5-mini can exceed GPT-5 in specific subcategories. The authors say these observations align with Table 2. Converted text cannot see colors; which specific sentences are marked red depends on the caption and authors' description.

Other ablations and analyses

The paper has no separate ablation table. Table 2 comparisons not itemized above:

• GPT-5-nano (high) overall 24.95, GPT-5-nano 23.07; both Contra. 5.24.
• o4-mini (high) Contra. 3.33, o4-mini 1.90; Inexist. 14.53 and 12.21, respectively.
• Qwen2.5-VL contradictory: 72B 19.05, 32B 1.43, 7B 1.90, 3B 0.00.
• InternVL2.5 overall: 78B 21.47, 38B 19.40, 8B 12.34, 4B 12.71; all four Contra. 0.00.
• GPT-4o-2024-11-20 overall 32.49, with Desc. 40.47, Open 20.73, Contra. 6.67.
• In the lowest open-ended tier, o4-mini (high) 9.52, Qwen2.5-VL-3B 7.84, InternVL2.5-4B 8.12.

The authors list three aspects of future work in the Limitations section; the experiments themselves stop at classification scores and qualitative cases for 15 models.

Limitations and future directions noted by the authors

• Data scope (Limitations): Data comes from scientific charts in CharXiv, mainly from arXiv papers. The authors state it cannot cover chart types from domains such as commercial, news, or education, thus limiting generalizability. Future direction is to expand data with charts from broader domains.
• Taxonomy coverage (Limitations): The authors state the taxonomy includes many scenarios that empirically trigger hallucination, but may not cover all cases in real usage; users may ask more diverse, context-dependent, or adversarial questions. The taxonomy can be further expanded in the future.
• Mitigation methods not evaluated (Limitations): The authors state this paper mainly diagnoses hallucination behavior and does not systematically incorporate or compare mitigation techniques. Future work can integrate and evaluate adversarial training, calibration, or abstention.

Experiment coverage

• The tested models are the 15 LVLMs listed in Section 4.1; Table 2 additionally reports scores for some of these models when reasoning effort is set to high.
• The evaluation data is CHART HAL's 1,062 items, with 12 categories distribution in Table 1. Charts are sampled from the CharXiv validation set, keeping only those with 1 or 2 subfigures, in a ratio of 3:2 (Appendix A.1).
• Model outputs are scored by GPT-4o-2024-11-20 using a binary rubric per Appendix A.2. The paper does not report the number of repetitions, nor the agreement between this automatic scoring and human judgment.
• Questions and answers are generated by o4-mini, then verified by annotators described in the appendix as holding a bachelor's degree; the main text Section 3.2 calls these annotators expert annotators. The paper does not report the number of annotators or the discard rate. The appendix states sampling was proportional across disciplines and covered multiple chart types, but does not report counts per discipline or chart type.
• At generation time, only the chart and question are input, and the sampling temperature is set to 0; Appendix B states that reasoning models like o4-mini cannot have temperature configured, and generation length uses each model's default limit. Figure 2 shows qualitative cases.

CHART HAL is a fine-grained benchmark for LVLM chart understanding hallucination.

• Problem: Chart QA requires strict consistency with the chart. The authors state that hallucination evaluation rarely enters charts, and chart benchmarks rarely break down how hallucination occurs.
• Approach: Free-form answers. Questions are divided into description, reasoning, and open-ended, and their relation to the chart into unrelated, absent, contradictory, and normal, totaling 12 categories. Charts are taken from CharXiv, keeping only those with 1 or 2 subfigures; o4-mini generates Q&A, which is manually revised, totaling 1,062 items. No hallucination is scored as 1, scored by GPT-4o-2024-11-20. Open-ended normal questions look for uncertain tone and reasonable reasoning, not factual correctness.
• Results: In Table 2, Qwen2.5-VL-72B-Instruct scores 54.24% overall, Gemini-2.5-Pro 49.34%, GPT-5 (high) 34.46%, o4-mini (high) 22.79%, Qwen2.5-VL-3B-Instruct 8.57%. The abstract attributes 34.46% and 22.79% to GPT-5 and o4-mini, which are the same as the high rows. Several open-source models score 0.00% on the 210 contradictory items. Gemini scores higher on Contra. and Normal than Qwen72B, which has the highest overall score.
• Author conclusion: Answerable questions are somewhat handled; when asked about content not in the chart or contradicting it, models often fabricate rather than admit unanswerability. The authors believe that increasing reasoning effort has unstable benefits because extra reasoning amplifies errors when the chart is misread. The authors also note that the data mainly comes from arXiv scientific charts, and both the taxonomy and mitigation evaluation can be expanded.