Microsoft open-sources CAVEAT, a benchmark for evaluating shopping agents.

1. 11:00
   论文追踪Selected

   Microsoft Research released CAVEAT: e-commerce steering mechanisms reduced the agent's optimal purchase rate from 78.6% to 17.3%

   Microsoft Research proposes the CAVEAT benchmark, using nine high-fidelity e-commerce environments and eight types of guidance mechanisms from commercial practice, to evaluate whether computer-using agents still select the user's optimal product when there are incentive conflicts between the environment and user goals. Across 52 standard tasks and 18 configurations from five model families, the optimal purchase rate is 78.6% when guidance mechanisms are disabled, dropping to 17.3% when all are enabled, with a reference degradation of 61.3 percentage points; CAVEAT-Hard expands the catalog to 2,112 products and 88 pages of results, with GPT-5.6-Sol-high dropping from 90.0% to 0.0%, stopping on page 1 in all 50 attempts. Trajectory analysis and ablations attribute failures to three categories: prematurely ending search, user-unstated preference ordering (preferences mentioned earlier in the request are treated as priorities), and placing orders before confirming key facts such as price. The researchers also used synthetic environment trajectories for two-stage post-training of Qwen3.5-27B to obtain CAVEAT-27B, achieving 22.9% on 16 held-out tasks when paired with Harness.

1. Date not given
   microsoft/CAVEATSelected

   Microsoft released CAVEAT, a shopping agent evaluation environment, where under steering, the optimal selection rate dropped from 78.6% to 17.3%

   Microsoft open-sourced CAVEAT for evaluating whether web agents follow user-established preferences and make optimal choices, with the only reported metric being the optimal selection rate. The evaluation covers nine fictional e-commerce and service environments, divided into CAVEAT-Standard (nine environments, 312 runs) and CAVEAT-Hard (five 2112-product scenarios, 10 runs per model/framework), with a BrowserUse baseline and an improved CAVEAT-Harness, supporting any OpenAI-compatible endpoint and custom harnesses. The repository documentation provides installation and running steps for Python 3.10+ with Chromium, and the paper is titled "CAVEAT: Towards Robust Computer-Use Agents in Incentive-Misaligned Environments," with the CAVEAT-27B model checkpoint to be released later.

Current attention59·Peak in the comparable period63（Oct 8 20:00）·Change over the comparable past 24 hours -2%