Old Ideas, New Problems: The Instability of LLM-Based Novelty Evaluation

The authors conducted a systematic controlled study of LLM-based novelty judgments and found that a small change in prompt wording alone can flip the novelty judgment for the same pair of ideas by more than half, with pairwise accuracy changing by more than 50 percentage points. The study automatically mined from OpenReview passages in which reviews explicitly affirmed or questioned a paper's originality, retained only submissions on which opinions were consistent at the extremes within their research field, and then paired them with ideas generated by ordinary LLMs. Across six judge models, merely telling the judge that reviewers considered one of the ideas novel and the other not novel changed its judgment on more than half of the same idea pairs, occasionally pushing accuracy below random level; the same change helped one judge and harmed another. Retrieval and larger reasoning budgets helped only limitedly, and two purpose-built novelty evaluators performed worse than the cheapest prompting baseline. On this basis, the authors question the novelty improvements reported by automated ideation systems.