When figures and text disagree: evaluating autonomous visual evidence verification in scientific papers
When Figures Disagree with Text: Evaluating Autonomous Visual Evidence Verification in Scientific Papers
The authors had six multimodal models autonomously review 31 papers containing figure–text contradictions: 60.6% of reviews mentioned the problematic figure, and only 3.3% were accepted as diagnosing the contradiction.
- Existing image-text consistency evaluations take the image, the claim, or
- From PRISMM-Bench, 31 complete papers containing reviewer-recorded image-text contradictions were taken; six multimodal models wrote reviews under a neutral prompt, scored on three nested levels: mention, questioning, diagnosis; then single inputs such as naming the figure, consistency instructions, highlighting, and cropping were added one per arm, and a two-stage baseline in which the model selects candidates itself was also tested.
- Of 1,445 open-ended reviews, 60.6% mention the flawed figure and 3.3% are accepted as diagnoses; in the controlled experiments, naming the figure raises S3 from 4.3% to 20.4%, consistency instructions and visual hints are ineffective, and the model selecting candidates by centrality itself yields only 4.8%.
- The authors state that the rubric is lenient (human blind review confirms only 52.6% of accepted diagnoses), that the controlled arms have no human audit, and that conclusions are limited to 31 image-heavy papers and six models; the two-stage results constrain only one candidate selection rule.
- Evaluated models
- Claude Sonnet 4.6GPT-5.4GPT-5 miniGPT-5 nanoGemini 3.1 ProGemini 3.1 Flash-Lite
- Benchmarks
- PRISMM-Bench(31篇完整论文及其审稿人记录的图文不一致)
- Metrics
- S1 mention(提及率)S2 concern(质疑率)S3 rubric-accepted diagnosis(评分规则接受的诊断率)shortlist recall(候选召回率)
The researchers propose an evaluation setting for autonomously verifying figure-text consistency in complete scientific manuscripts, having six multimodal models review 31 complete manuscripts containing reviewer records and figure-text contradictions hidden from the models, with the contradictions coming from PRISMM-Bench. Of 1,445 open-ended review comments, 60.6% mentioned the flawed figure, but under the rubric only 3.3% were accepted as diagnosing the contradiction; blind review found the rubric lenient, so the authors treat these proportions as rubric acceptance rates, used for comparison between conditions rather than absolute levels. Controlled experiments show that naming the figure without disclosing the flaw raises the rubric-accepted diagnosis rate from 4.3% to 20.4%, whereas consistency instructions and coarse visual hints are ineffective; when the contradiction is visible only in the rendered figure, text-only review obtains no accepted diagnosis, while naming the figure and reviewing the rendered page reaches 15.8%; the baseline that self-selects figures to check by centrality stalls at 4.8%. The authors therefore conclude that deciding what to check is the main difficulty, which their tested selection rules fail to solve, and that multimodal document understanding should evaluate whether a model can propose what to check, not merely answer questions about a given figure.
Paper analysis
What problem does this paper attempt to solve?
Evaluates whether multimodal models can autonomously discover, align, and verify image-text contradictions in complete papers, rather than answering questions about a given figure.
Core question: Can multimodal models, in a complete paper without any hints, autonomously find and diagnose figure–text inconsistency?
- Setting and importance: The authors argue that scientific figures are evidence for claims cited in the text, and that reviewing a paper is not just describing the figure but also checking whether the figure and text are consistent; multimodal models have been used to write feedback on real manuscripts, but fluently describing a figure does not mean it was ever compared with the text.
- Shortcomings of existing evaluations: The authors state that existing benchmarks put figures, claims, or
What related research exists?
Related work is organized by
Claim and evidence are given
- ChartCheck (2024) — interpretable fact-checking on real chart images.
- MuSciClaims (2025) — pairs claims with figures and their captions; the paper notes that its diagnosis task shows models are poor at locating supporting evidence in figures and tend to judge claims as supported.
- SciVer (2025) and SPIQA (2024) — respectively expert-annotated claim verification on the text, tables, and figures of a single paper, and question answering about paper figures. The paper notes that because the claim and figure are given, such benchmarks cannot test whether a model can itself select the claim and figure that should be checked.
The existence of the inconsistency is given
- MMIR (2025) — injects synthetic inconsistencies into web pages, slides, and posters; the paper relays its finding that models handle conflicts between two elements better than conflicts within a single element in complex layouts.
- PRISMM-Bench (2026) — collects 384 reviewer-flagged inconsistencies across text, figures, tables, and formulas, and tests identification, repair, and pairing with multiple-choice questions; the paper notes that even when the question states an inconsistency exists, model scores remain low. This paper reuses its recorded inconsistencies but switches to complete papers with a neutral review request, no options, and restores one hint per arm.
The target is given in the query
- Chart/document QA and long-document benchmarks — ChartQA, DocVQA, CharXiv, MMSci, MMLongBench-Doc, etc., ask questions about a given figure or page.
- Localization benchmarks and studies — SeeClick and ScreenSpot-Pro test small-target localization on dense screens, and Visual Haystacks tests finding clues across multiple images; Eyes Wide Shut? (2024) reports systematic failures of models on basic visual patterns, and hallucination benchmark studies report errors about content that does not exist. The paper notes that in this paper's setting the content exists but the contradiction is not reported, the opposite of hallucination.
Only the paper is given (automatic review)
- PeerRead (2018) collects a review corpus; Liang et al. (2024) and ReviewerGPT? (2023) test whether model feedback is useful; Thakkar et al. (2026) conducted a large-scale randomized study of model feedback; Zhang and Abernethy (2025) have reasoning models find a paper's key problems. The paper notes that these works judge what a review says overall, and a fluent review can discuss figures without comparing them to the text, whereas this paper checks whether this specific behavior occurs within each review.
Positioning: the authors state that this paper differs from the above work in that nothing is given — only the complete paper — and what is tested is whether the model can itself propose what to verify.
How does the paper address this problem?
Using 31 complete papers containing contradictions recorded by reviewers, six models were asked to write reviews under a neutral prompt, scored on a nested four-level outcome, and then a single input was changed for each arm.
Overall approach: hide the real image-text inconsistencies recorded by reviewers in PRISMM-Bench inside complete paper PDFs, have models write free-form reviews under an autonomous verification setting, score by how far the review gets; then change only one input at a time to see which change pushes the review further.
Task and data
- Unit of study: a complete paper paired with one inconsistency recorded in PRISMM-Bench, which specifies a visual element (figure or table) and the claim that conflicts with it; the model sees the rendered pages and a neutral review request, is not told that an error exists or where it is, and the recorded inconsistency is used only for post-hoc scoring.
- Paper collection: 31 papers containing figures, selected according to a plan frozen before review reading; longer than the benchmark candidate pool (median 17 pages vs. 15), and all conclusions are limited to this image-heavy set (Appendix A: 29 with figure-only issues, 0 with table-only issues).
- Three research questions: RQ1 where unguided review stops; RQ2 which single input change (naming the figure, requiring a consistency check, highlighting, removing the visual channel) advances diagnosis; RQ3 whether the model can select candidate figures itself. RQ3 is a diagnostic baseline, not a proposed method.
Four-level nested outcomes and scoring
- Outcome definitions: a review may miss the flawed element, mention it, question it, or diagnose the recorded contradiction. L1 marks reviews that name the target figure/table, L2 marks reviews that name it and question it in the same sentence, and P marks reviews that both scoring models accept as diagnosing the recorded contradiction with quoted text.
- Nested rates: S1 = L1 ∪ P (mention), S2 = L2 ∪ P (questioning), S3 = P (rubric-accepted diagnosis). The authors stress that these rates describe what the review says and do not separately measure the three internal steps of candidate discovery, alignment, and verification.
- Scoring and audit: Gemini 4 Argon and GPT-5.4 double-score without seeing the model or condition, with κ=0.62 on 1,221 decidable pairs; another pair of models (GPT-6 Astra, Claude Opus 5) replayed the rubric, with an acceptance rate of 2.74% vs. 3.28% for the main scoring; two authors blind-reviewed 100 open-ended reviews, and of 19 decidable rubric acceptances only 10 (52.6%) were confirmed by both, so the authors state the rubric is lenient and all rates are to be read only as rubric acceptance rates, comparing conditions rather than reading absolute values. The 175 undecidable reviews remain in the denominator and are counted as not diagnosed.
Experimental conditions
- Open-ended queue: 1,445 reviews = 31 papers × 6 models × 2 repetitions × 4 input conditions (original pages, byte-identical repeat, flawed figure with a border, unrelated figure with a border); the prompt does not name the figure or mention the error, but two conditions add a visual hint, so the pooled queue is not unprompted.
- Controlled arms: subsequent runs on the same 31 unmodified PDFs, 186 reviews per arm (31×6). The unprompted arm reuses the queue prompt; each of the other arms adds one component: target figure identity (without stating there is an error), an instruction to check for contradictions, a crop of the flawed element placed in different positions, a crop plus a conflict clause; there are also a text-only arm (PDF text layer only, figures named by placeholders) and an OCR-only arm (text inside figures only, no rendered traces).
- Two-stage baseline (RQ3): Stage 1 reads the unmodified PDF, selects 5 figures/tables by centrality to the paper's claims and records each one's caption claim and body-text claim, then ranks 3 pairs for verification, explicitly told not to report errors; Stage 2 hands the top three and their page numbers back to the model along with a consistency instruction to write one review. Two-stage token usage is 2.04 times that of a single pass; units in Stage 1 with no usable identifier are counted as not diagnosed.
- Statistics: intervals are 95% bootstrap intervals resampled by paper (10,000 draws), and paired comparisons use per-paper differences and exact sign tests.
What experiments did the paper conduct?
In open-ended review, 60.6% mentioned the flawed figure, and 3.3% were accepted as diagnosing it; naming the figure raised this to 20.4%, consistency instructions and visual prompts were ineffective, and self-selected candidates were only 4.8%.
Experimental setup
- Models tested (reviewers): Claude Sonnet 4.6, GPT-5.4, GPT-5 mini, GPT-5 nano, Gemini 3.1 Pro, Gemini 3.1 Flash-Lite, 6 in total.
- Data: 31 complete papers, each hiding one reviewer-recorded image-text inconsistency from PRISMM-Bench; the open-ended queue has 1,445 reviews (31×6×2 repetitions×4 conditions), and the controlled runs have 186 reviews per arm (31×6).
- Scoring: Gemini 4 Argon and GPT-5.4 double-score the rubric (κ=0.62, 1,221 pairs); GPT-6 Astra and Claude Opus 5 independently replay; two authors blind-review 100 open-ended reviews (covering only the open-ended queue).
- Metrics: S1 mention, S2 questioning, S3 rubric-accepted diagnosis; intervals are paper-clustered 95% bootstrap CIs.
- Repetitions and sample size: the open-ended queue has 2 repetitions per model per paper; each controlled arm has 1 review per model per paper (31 per model); the per-model controlled numbers are based on 31 reviews, which the authors call descriptive.
Main results
Condition S3 (rubric-accepted diagnosis) Source Open-ended queue (1,445 reviews, 4 conditions pooled) 3.25% [1.60, 5.18] Section 4.1, Table O.1 Controlled · unprompted (186 reviews) 4.30% [1.08, 8.06] Table 2 Controlled · global consistency instruction 6.99% [2.69, 12.37] Table 2 Controlled · naming the target figure 20.43% [13.44, 28.49] Table 2 Controlled · naming the figure + consistency instruction 17.20% [10.22, 25.27] Table 2 Controlled · two-stage self-selected candidates 4.84% [1.61, 8.60] Table J.1 - In the open-ended queue, 39.4% miss the flawed element, 34.2% only mention it, 23.2% raise a question that is not accepted, and 3.3% (47 reviews) are accepted as diagnoses; the nested rates are S1=60.6% and S2=26.4% (Section 4.1).
- The authors state that naming the figure raises the diagnosis rate by +16.13 pp [+9.14, +24.19], while all intervals for the consistency instruction include zero; after naming the figure, about four-fifths of reviews still do not diagnose the contradiction.
- The authors state that the two-stage baseline is only +0.54 pp [−3.76, +5.38] over unprompted and −15.59 pp below naming the figure, recovering only 3.3% of the gain from naming the figure (the interval [−33.30, +27.30] is too wide to rule out any extreme).
RQ1: Where review stops
- What was tested: the four-level outcome distribution of 1,445 open-ended reviews, the mention comparison between the flawed element and its nearest same-type unflawed neighbor, and the effect of border highlighting.
- Results: of the 47 accepted diagnoses, Claude Sonnet 4.6 contributes 37 (15.3% of its 242 reviews), and the other five models together contribute 0.83% [0.17, 1.65], even though 55.4% of their reviews mention the flawed element; 1 paper contributes 10 diagnoses, and excluding it gives 2.65%. The flawed element's mention rate is 57.9% vs. 53.6% for neighbors (+4.3 pp [−5.5, +14.0], 1,309 pairs). A border on the flawed figure relative to a border on an unrelated figure raises the questioning rate by +5.6 pp [+0.8, +10.5] and diagnosis by +1.1 pp [−0.4, +2.7], while the per-paper mean absolute fluctuation for byte-identical repeats is 11.6 and 3.5 pp, and the authors state that both movements are within run-to-run fluctuation.
- Near-miss analysis: of 637 near-miss records, 82.1% do not match the recorded element, 16.8% match the element but not the relation, and 1.1% match both (Figure 2c); in blind review, 21 of 30 element-mismatch records were considered by at least one reader to indeed criticize the target element, and the authors state that the first label overestimates misses of the figure itself.
RQ2: Which intervention advances diagnosis
- Per-model gains (Figure 2b, 31 papers per model): after naming the figure, Claude Sonnet 4.6 rises from 16.1% to 45.2% and Gemini 3.1 Pro from 9.7% to 35.5%; GPT-5.4, Gemini 3.1 Flash-Lite, GPT-5 mini, and GPT-5 nano go from 0% to 19.4%, 16.1%, 6.5%, and 0% respectively; the five models pooled rise from 1.9% to 15.5%.
- Change in the nature of misses: after naming the figure, the proportion of near-misses that do not match the recorded element falls from 78.4% to 55.8%, and the proportion that match the element but not the relation rises from 9.8% to 41.8% (Figure 2c); the authors state that reviews often reach the correct element but state a different relation.
- Providing the conflict clause: on the 14 papers with a concrete contrast, the crop plus conflict clause diagnoses 15.5% [4.8, 29.8], vs. 2.4% unprompted and 22.6% for naming the figure alone; the difference from naming the figure is −7.1 pp [−23.8, +10.7], and the authors state that with 14 papers this estimate is uninformative.
- Crop placement and no-reference panel: placing the crop next to the caption, elsewhere on the page, or in the band containing the contradiction gives diagnoses of 3.2%, 1.6%, and 3.2% respectively, close to the 4.3% unprompted. The no-reference panel (without seeing the answer) also finds the asserted conflict in 36.0% of text-only reviews, while text-only diagnosis is only 2.2%; the authors state that an asserted conflict is not the same as diagnosing the recorded contradiction.
RQ3 and breakdown by evidence type
- Candidate recall: the proportion of Stage 1 top-three shortlists containing the flawed figure is 34.41% [23.66, 46.24], and for top five it is 57.53% [43.55, 71.51]; when the top three do not contain the flawed figure, diagnosis is 4.1% (122 runs). The authors state that ranking by centrality cannot reliably surface figures that contradict the text.
- Exploratory subset: among the 64 runs whose shortlist contains the flawed figure, the three-figure batch check diagnoses 6.25% [0.00, 13.85], naming the figure plus a consistency instruction on the same 64 runs gives 20.3% (paired difference −7.5 pp [−13.9, −2.3]), and naming the figure alone gives 15.6%, a difference of −1.5 pp [−9.9, +9.7]; the authors state that this subset is defined by Stage 1 results and was not preplanned, and the comparison is exploratory.
- Breakdown by evidence type (Figure 3c): on the 20 papers where the evidence is only in the figure, text-only diagnosis is 0/120, unprompted is 1.7%, and naming the figure is 15.8% [8.3, 24.2]; on the 11 papers where the evidence is also in the text, these are 6.1%, 9.1%, and 28.8% [13.6, 45.5] respectively. The authors state that this matches the task design's expectation but cannot show that the gain from naming the figure itself depends on pixels (the text-only + naming-the-figure arm is missing), and the stratified intervals are wide.
Other ablations and analyses
- Per-model open-ended S3 (Table C.1): Claude Sonnet 4.6 is 15.29%, Gemini 3.1 Pro is 2.90%, GPT-5.4 is 0.83%, Gemini 3.1 Flash-Lite is 0.42%, and GPT-5 mini and GPT-5 nano are 0.00%.
- Excluding Claude Sonnet 4.6, the queue S3 falls to 0.83% [0.17, 1.65] (10/1,203, Appendix C).
- Breakdown by problem type (Table O.1): Type-I literal mismatch (13 papers) has S3 of 4.94%, and Type-II interpretive dispute (18 papers) has 2.03%, a difference of +2.91 pp [−0.82, +7.00]; the authors state that the interval includes zero and this is only suggestive.
- Independent replay acceptance rate is 2.74% vs. 3.28% for the main scoring (1,279 pairs, paired difference −0.55 pp); the source-anchored three-model panel agrees with the rubric on 44 items at 79.5% three-way agreement (κ=0.556), and on decidable binary classifications κ=0.906 (Appendix I).
- Human blind review: the two authors have κ=0.898; of 19 decidable rubric acceptances, 10 (52.6%) are confirmed by both, and no rejected review is confirmed by both (Section 3).
- After Holm–Bonferroni correction, the smallest pHolm for the border comparison is 0.767, and on S3 each visual intervention moves at most +1.11 pp (Table O.3).
What are some points that can be further explored?
The authors state that the rubric is rather lenient, the controlled arms had no manual audit, and the conclusions are limited to 31 figure-centric papers; next steps could test ranking by contradiction propensity or verifying each candidate individually.
Limitations noted by the authors and future directions
- The rubric is lenient: blind review found that the rubric was lenient in a considerable proportion of accepted reviews, so absolute rates may be inflated; the controlled arms had no human audit, and differences between arms reflect differences in the confirmed diagnosis only if the rubric error is similar across arms, which the authors did not test (Section 3, Section 5 Scope).
- Sample and coverage: conclusions are based on 31 image-heavy papers that are longer than the candidate pool, 6 models, and a single model-based rubric; one model contributed most of the open-ended diagnoses, and the per-model controlled numbers are based on only 31 reviews (Section 5 Scope).
- Exploratory analyses: the paired-excerpt arm, the breakdown by evidence type, and the shortlist-subset comparison are all exploratory (Section 5 Scope).
- Only one selection rule is constrained: the two-stage results constrain only the centrality ranking rule tested; rules that rank by contradiction propensity, or that verify each candidate individually, were not tested, and the authors call this a natural next step on this protocol (Section 4.3, Section 5).
- Internal mechanisms cannot be localized: the arms change only the input, not the internals, so they show where review stops rather than why; they cannot distinguish whether alignment or verification is the step that limits the model (end of Section 4.2, Section 5).
- Three points from the appendix self-check: all checks are verification rather than discovery (the paper collection comes with answers, and whether models can discover unlabeled inconsistencies was not tested); the consistency numbers are agreement with the registered answers, and human blind review showed that about half of accepted reviews do not carry the diagnosis the rubric assigned, with the bias direction being over-counting; these checks constrain the measurement instrument rather than the construct itself (Appendix O).
Scope of experiments
- There are 6 reviewer models tested: Claude Sonnet 4.6, GPT-5.4, GPT-5 mini, GPT-5 nano, Gemini 3.1 Pro, Gemini 3.1 Flash-Lite (Section 3).
- The paper collection has 31 papers, all from the PRISMM-Bench candidate pool: 29 with figure-only issues, 0 with table-only issues, and 2 with both figures and tables (Appendix A, Table A.3).
- The open-ended queue is 2 repetitions per model per paper; each controlled arm has 186 papers (31×6), and 31 papers per model; the paired-excerpt arm has 84 papers and 14 papers (Table 1).
- Human blind review covers only the 100 reviews of the open-ended queue; the controlled arms have no human audit (Table 1).
- The paper does not report query cost or time per controlled arm; the token usage of the two-stage baseline is 2.04 times that of a single pass (Section 4.3).
Summarize the main content of the paper
Six multimodal models autonomously reviewed 31 papers containing figure–text contradictions: 60.6% mentioned the flawed figure, and only 3.3% were accepted as diagnosing it; naming the figure raised this to 20.4%, and self-selected candidates were ineffective.
An evaluation paper on multimodal document understanding, studying whether models can autonomously verify figure–text evidence in complete papers, rather than answering questions about a given figure.
- Problem: Existing benchmarks give the figure, claim, or