Skip to content
Ai2·Indexed by this site · Originally published· AI updates →· Its place in the scan record →

Ai2 releases XYBench: testing whether models can solve the real problem behind the question

XYBench: Helping with the problem behind the question

AI summary

Ai2, in collaboration with Carnegie Mellon, NVIDIA and Johns Hopkins, released the benchmark XYBench, containing 8,115 real questions from sources including Stack Overflow, Stack Exchange and WikiHow, to test whether AI assistants can identify misconceptions in a question and redirect to the user's true goal; it won the EMNLP 2026 Best Resource Paper Award. Evaluations of Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro show that all five models are more inclined to answer the literal request; on Stack Overflow questions, answers focused on the underlying goal accounted for only 35–41%, and the proportion that guided users to abandon the wrong approach was 47–60%, lower than the 79% of human reference answers. The benchmark currently covers only English and programming-related questions.

Read originalallenai.org