Can we trust LLMs to judge whether research ideas are novel?
Can we trust LLMs to judge whether a research idea is novel?
Ai2 released the Old Ideas, Novel Problems study along with a supporting benchmark, data and code, examining the reliability of LLM judges in assessing the novelty of research ideas. The benchmark mines reviewers' discussions of originality from ICLR 2026 peer review to construct 154 high-novelty and 145 low-novelty human research ideas, and additionally has a simple generator produce 154 AI ideas to form a human-AI paired evaluation, testing six judge models: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6. In the human-AI paired evaluation, GPT-5.4's accuracy dropped from 93.5% to 40.9% due solely to a prompt change, below the random level; the same change also caused Claude Opus 4.6 to fluctuate by 21 percentage points on the human ideas dataset.
Tests of six LLM judges show that changing a few words in the prompt can cause large fluctuations in novelty judgments; teams using such judges to screen AI-generated research ideas can use this to check their evaluation setup.