Microsoft released CAVEAT, a shopping agent evaluation environment, where under steering, the optimal selection rate dropped from 78.6% to 17.3%
微软研究院 CAVEAT：电商平台一开赞助位、滴灌定价等引导手段，购物 agent 买到用户最优商品的比例从 78.6% 掉到 17.3%（代码与环境）
Microsoft open-sourced CAVEAT for evaluating whether web agents follow user-established preferences and make optimal choices, with the only reported metric being the optimal selection rate. The evaluation covers nine fictional e-commerce and service environments, divided into CAVEAT-Standard (nine environments, 312 runs) and CAVEAT-Hard (five 2112-product scenarios, 10 runs per model/framework), with a BrowserUse baseline and an improved CAVEAT-Harness, supporting any OpenAI-compatible endpoint and custom harnesses. The repository documentation provides installation and running steps for Python 3.10+ with Chromium, and the paper is titled "CAVEAT: Towards Robust Computer-Use Agents in Incentive-Misaligned Environments," with the CAVEAT-27B model checkpoint to be released later.
CAVEAT 把电商引导手段做成可复现的本地评测环境，做 Agent 购物与安全评测的团队可以按文中的配置直接跑出自己的最优选择率。