Ai2 and research collaborators have introduced XYBench, an evaluation designed to test whether an AI assistant recognizes when a user’s requested solution rests on a mistaken assumption. Their November 13 overview asks a practical question: does an assistant merely answer what was asked, or help with the underlying problem?
The problem behind the request
The benchmark uses the familiar “XY problem”: someone wants to accomplish X, assumes Y is the right approach and asks how to do Y. A useful answer may need to explain why Y is a poor fit and propose another path, rather than confidently optimizing the wrong approach.
The researchers describe 8,115 examples spanning programming and everyday procedural questions. The mix matters: 1,557 come from Stack Overflow, 184 from Stack Exchange, 102 were manually curated and 6,272 were generated from WikiHow guides. It would be misleading to describe the entire set as questions directly asked by real users.
How the results were judged
The team reports that five evaluated models often included useful alternatives but tended to emphasize the literal request more than the underlying goal. They used Claude Sonnet 4.5 as an automated judge and compared its classifications with assessments from two human reviewers on a subset of 500 responses. Reported agreement varied by metric; emphasis was less consistent than detecting whether a suggested solution appeared.
Those results are the researchers’ findings in a single-turn, English-language evaluation, not a universal ranking of assistants or proof of performance in an ongoing conversation. A model can also misread an actual user’s goal, making clarification important when the stakes or ambiguity are high.
The dataset is available for inspection; check its current terms before reuse. We have not independently run XYBench or reproduced the reported results.
Original reporting based on linked sources; no independent product or benchmark test.
