Skip to content
EventPast event

OpenDS-World benchmark: 12 frontier models score at most 39%

1 reports1 reporting sourcesLatest report

Start here

AI overview

A team from the Hong Kong University of Science and Technology (Guangzhou), Fudan University and other institutions proposes the OpenDS-World benchmark in an arXiv preprint, containing 200 real-web research tasks: each task starts from a video clue and requires the agent to discover and connect evidence across real websites, with an independently verified evidence chain and a unique factual answer. The best of the 12 tested frontier models answers only 39% correctly, and the paper states that this does not constitute a general ranking of research assistants.

Generated by AI from reports · Updated 4 days ago

Attention and developments

Attention, hourly

Now 34, 0% in 24 hours·Peak 37 (Nov 15 21:00)

Nov 16Nov 17Nov 18Nov 19Nov 20Now 34

Storyline

Developments on the same subject and issue, from earliest to latest

2 events
  1. · 1 reports· Attention38
    OpenDS-World benchmark released, 12 models top out at 39% accuracy
  2. · 1 reports· Attention34

    OpenDS-World benchmark: 12 frontier models score at most 39%This event

Progress timeline

Nov 15
  1. X @jiqizhixin (Synced)
    Among 12 frontier models on OpenDS-World's 200 real-web research tasks, the best result was only 39%

    A team from institutions including the Hong Kong University of Science and Technology (Guangzhou), Fudan University, the University of Washington, and Imperial College London proposed the OpenDS-World benchmark in the arXiv preprint Deep Search in Real-World Web Environments; among the 12 frontier models tested, the best result was only 39% of 200 real-web research tasks answered correctly. Each task starts from a video clue and requires the agent to discover, access, and connect evidence across real websites, with information possibly hidden in interactive pages, audiovisual materials, scientific charts, or structured files; tasks come with independently verified evidence chains and unique factual answers, rather than preset tool-call sequences. All models share the same tool environment, with a search budget of 3600 seconds or 100 tool calls, whichever is exhausted first. The text states that this accuracy reflects only this benchmark and execution protocol and does not constitute a general ranking of research assistants. The paper is at https://arxiv.org/abs/2610.30181,代码开源于 https://github.com/Yu-xm/OpenDS。

Related discussion

Total 1 items

Attention trend

New sources per day

2 in 6 days·most 2 (Nov 15)·today 0

  • Nov 15: 2 new sources (1 outlets, 1 accounts)
  • Nov 16: 0 new sources (0 outlets, 0 accounts)
  • Nov 17: 0 new sources (0 outlets, 0 accounts)
  • Nov 18: 0 new sources (0 outlets, 0 accounts)
  • Nov 19: 0 new sources (0 outlets, 0 accounts)
  • Nov 20: 0 new sources (0 outlets, 0 accounts)

Hourly attention

Current attention 34·Peak in the comparable period 37 (Nov 15 21:00)·Change over the comparable past 24 hours 0%

010203040Nov 1517:00Nov 1704:00Nov 1815:00Nov 2002:00