AI & SocietyAI & Society · 一项横断面研究评测六款生成式AI聊天机器人的心脏骤停公众咨询表现。研究用56个心脏骤停相关公众咨询问题，在2026年5月10日至16日间对ChatGPT Plus 5.5 thinking、Gemini 3.5 Flash、Microsoft Copilot smart、DeepSeek-V4-pro、Doubao和Perplexity各提问一次，共获得336条回答，由五名盲评心内科专家从安全性、准确性、共情、DISCERN、EQIP、JAMA和GQS等维度评估。
论文追踪, microsoft/CAVEAT论文追踪 · 2 sources · 2 reports · Microsoft open-sources CAVEAT, used to evaluate whether web agents follow users' stated preferences and make optimal choices; the only reported metric is the optimal choice rate. The benchmark covers nine fictional e-commerce and service environments, split into CAVEAT-Standard (nine environments, 312 runs) and CAVEAT-Hard (five scenarios with 2,112 products, 10 runs per model/framework), and comes with a BrowserUse baseline and an improved CAVEAT-Harness, supporting any OpenAI-compatible endpoint and custom harnesses. Under steering techniques, the optimal choice rate drops from 78.6% to 17.3%. A follow-up paper explains that the benchmark uses nine high-fidelity e-commerce environments and eight categories of steering mechanisms drawn from commercial practice to evaluate whether computer-using agents still choose the user-optimal product when the environment and user goals have conflicting incentives; across 52 standard tasks and 18 configurations from five model families, the optimal purchase rate is 78.6% with steering disabled, dropping to 17.3% with all steering enabled, a degradation of 61.3 percentage points; CAVEAT-Hard expands the catalog to 2,112 products and 88 pages of results, where GPT-5.6-Sol-high drops by more than 90 percentage points.
