Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories

A large-scale study covering 6,959 paper revision trajectories shows that AI reviewers' scores remained unchanged on most substantive revisions, and brief negative user comments may offset or reverse the score gains brought by revisions. The research team built the PaperTrajectory-2026 dataset, covering the paper revision trajectories of five AI conferences, automatically annotated revision changes along 14 dimensions, and had eight AI reviewers independently score different versions of the manuscripts. The annotations show that major improvements were concentrated in empirical evidence and framing, while later revisions were mostly incremental improvements. The reviewers' consistency in judging the direction of score changes was weak, the association between revision dimensions and score increases varied by reviewer, and predictive discrimination was limited. The authors state that these findings indicate substantive revisions were not stably reflected in review scores, and that evaluation methods are needed that can connect manuscript evidence with overall judgments and remain stable when the scientific evidence is unchanged.