Ai2 tests AI research judges—and finds their novelty verdicts can swing

On October 27, 2026, Ai2 published a study and benchmark examining whether language models can reliably judge the novelty of research ideas. The researchers also released data and code. This article was published the same day.

What they found

Ai2 reports that in one experiment, small wording changes shifted a model’s accuracy on pairwise novelty judgments by more than 50 percentage points. The study also compared simpler prompts with more elaborate search-based systems on particular evaluation sets. These are findings from the researchers’ experiments, not a general ranking of every model or research workflow.

The benchmark dataset and linked paper make the setup available for inspection. Ai2 cautions that results can vary with the model, prompt and whether the ideas being evaluated are human-authored or AI-generated.

Why it matters

AI can suggest plausible directions quickly, but an apparently confident novelty verdict does not replace checking prior work. Teams using AI for research should compare outputs across prompts, consult original sources and involve people familiar with the field. AiBrainWorX has not independently reproduced these results.