AI agents have poor taste in long-horizon decisions, as demonstrated by the new benchmark Taste-Bench

The researchers release the Taste-Bench benchmark, which measures an agent's long-horizon decision judgment using binary choices at trajectory forks. Among the 14 tested models, the highest, GPT-5.6 Sol, achieves an average accuracy of 59.7%, GPT-5.5 achieves 59.5%, and the remaining models are significantly lower. The benchmark contains 502 questions, with pure guessing at 25%; when key clues are already visible in previous steps, the average accuracy across all models is 62.3%, dropping to 21% when clues appear only after substantial subsequent work, and increasing the reasoning budget provides almost no improvement. The researchers also distill the teacher model's judgment into Qwen3.6-27B, raising accuracy on unseen tasks from 30% to 47.9%, and improving the success rate from 14.6% to 33.7% on 41 held-out SWE-bench Pro tasks. The benchmark and code are publicly available.