OpenDS-World benchmark released, 12 models top out at 39% accuracy
Start here
A team from institutions including the Hong Kong University of Science and Technology (Guangzhou), Fudan University, and the University of Washington proposed in an arXiv paper a real open-web deep search benchmark, OpenDS-World, containing 200 tasks covering 4 modalities, 42 resource formats, and 7 categories of evidence access needs, using an approach that first constructs evidence chains and then reverse-generates questions. Among 12 evaluated frontier models, the strongest, GPT-6 Astra, achieved only 39.0% accuracy.
Generated by AI from reports · Updated 20 days ago
Storyline
- · 1 reports· Attention38
OpenDS-World benchmark released, 12 models top out at 39% accuracyThis event
- · 1 reports· Attention34OpenDS-World benchmark: 12 frontier models score at most 39%
Progress timeline
- WebSynced (Jiqizhixin)· Media coverageThe strongest model's accuracy is only 39%! When Deep Search truly enters the complex real internet(opens the original post in a new tab)
- WeChat · Synced (Jiqizhixin)OpenDS-World benchmark: Deep Search enters the real internet, strongest model GPT-6 Astra reaches only 39% accuracy
A research team from institutions including The Hong Kong University of Science and Technology (Guangzhou), Fudan University and the University of Washington proposed OpenDS-World, a deep search benchmark for the real open web, in the arXiv paper "Deep Search in Real-World Web Environments"; among 12 evaluated frontier models, the strongest, GPT-6 Astra, reached only 39.0% accuracy. The benchmark contains 200 tasks covering 4 modalities, 42 resource formats and 7 types of evidence access requirements, and uses an approach of first constructing evidence chains and then reverse-generating questions; models must search autonomously within a budget of 1 hour and 100 tool calls and give verifiable factual answers. Claude Opus 5.5 and Gemini 3.8 Flash achieved 38.0% and 36.5% respectively; 50 human participants made 1,000 independent attempts in total, of which only 211 were correct, and 85 questions were solved by no one. The paper proposes that the benchmark can be used to study agents' long-horizon decision-making and recursive self-improvement (RSI).
Related discussion
- Older than 72 hours · Not included in the 72-hour figures
- WebSynced (Jiqizhixin)The strongest model's accuracy is only 39%! When Deep Search truly enters the complex real internet(opens the original post in a new tab)