Microsoft Research released CAVEAT: e-commerce steering mechanisms reduced the agent's optimal purchase rate from 78.6% to 17.3%

When the online environment itself has interests, can delegated computer-use agents maintain user goals.

• Scenario and significance: Computer-use agents (CUAs) increasingly act autonomously on behalf of users across online platforms for tasks such as search, price comparison, and purchase decisions, but real-world platforms often advance their own interests through ranking, recommendations, and interface design.
• Limitations of existing evaluations: The authors note that existing capability benchmarks mainly test task completion in neutral cooperative environments, while safety benchmarks focus on explicit adversarial attacks (e.g., prompt injection); neither evaluates whether agents deviate from user goals when the environment operates normally but has its own preference.
• Core observation and formalization: The environment pursues its own goal V through steering mechanisms that manipulate information presentation order, salience, and disclosure timing, creating an incentive misalignment between the decision environment and the user goal U.
• Proposed approach: The paper introduces the CAVEAT benchmark, covering 9 high-fidelity online marketplace environments and 8 categories of commercial steering mechanisms, and quantifies the degradation of agent decisions in conflicted environments through paired controlled experiments.

The paper divides related work into three categories and clarifies its positioning relative to them:

• Computer use and shopping agents: Benchmarks such as WebArena (Zhou et al., 2024), Mind2Web (Deng et al., 2023), and OSWorld (Xie et al., 2024) evaluate navigation and task completion on real interfaces; WebShop (Yao et al., 2022), ShopperBench (Ling et al., 2026), and ShoppingBench (Wang et al., 2026) extend to product search and complex needs; Zhao et al. (2025) and Jiang et al. (2025) test preference following. The authors note that CAVEAT fixes user decisions and changes environmental incentives, testing end-to-end whether agents achieve user-optimal outcomes.
• Agent safety and deceptive interfaces: AgentDojo (Debenedetti et al., 2024), WASP (Evtimov et al., 2026), and InjecAgent (Zhan et al., 2024) study prompt injection; DECEPTICON (Cuvin et al., 2026), SusBench (Guo et al., 2026), Shi et al. (2026), and ABxLab (Cherep et al., 2026) test agent susceptibility to dark patterns. The authors note that CAVEAT does not inject false facts; all prices and attributes are truthful and verifiable, only altering the order, salience, and timing of genuine information, and evaluates final purchase decisions over the full catalog.
• Platform incentives and consumer choice: Research in economics, recommender systems, and human-computer interaction studies how platform goals shape display ordering and decisions (Xu et al., 2022; Mathur et al., 2019; Athey & Ellison, 2011). CAVEAT brings such incentive mechanisms into interactive embodied environments and defines verifiable optimal solutions.
• Baseline methods and benchmarks: Experiments use the open-source framework BrowserUse and its official system prompt as the base scaffolding, evaluated on CAVEAT-Standard and CAVEAT-Hard consisting of 9 self-built marketplace environments.

The authors position this work as the first study to evaluate, with paired causal contrasts, the impact of commercial steering mechanisms on end-to-end agent decisions in full multi-page marketplace environments while keeping facts truthful and tasks fixed.

The paper addresses this problem by formalizing incentive misalignment, building high-fidelity simulated marketplace systems, and designing counterfactual paired evaluations and intervention strategies.

Problem setup and formalization

Let the feasible outcome set be X, the user goal be U, and the user-optimal outcome set be X^∗_U = argmax_{x ∈ X} U(x). The agent interacts with the environment E and selects an outcome x̂_E. Decision quality is defined as the optimal purchase rate R(E), the proportion of rounds where the selected outcome falls within X^∗_U. If the environment's goal V favors outcomes outside X^∗_U, the environment is called an incentive-misaligned environment. For each misaligned environment E, the paper constructs a paired control environment E₀ (with steering mechanisms disabled) and defines the incentive-induced degradation metric as Δ = R(E₀) − R(E).

Marketplace environments and task construction

• 9 domain environments: Cover CAVEAT-Stay (short-term rentals), CAVEAT-Shop (retail), CAVEAT-Food (food delivery), CAVEAT-Grocery (fresh groceries), CAVEAT-Market (second-hand resale), CAVEAT-Kicks (sneakers), CAVEAT-Services (outsourcing), CAVEAT-Craft (handicrafts), and CAVEAT-Sport (sporting goods), all supporting the full flow of search, filtering, pagination, product details, and checkout.
• Authenticity and anti-scraping mechanisms: Frontends are built modeled on real platforms but do not use real brands to avoid pretraining contamination; request rate limits and a 6-digit CAPTCHA mechanism are set (triggered in about 1% of experiments).
• Task definition and unique optimal solution: User requests include hard constraints (e.g., budget caps) and comparative preferences (e.g., lowest price). Each task is strictly verified programmatically: in the full product catalog, there is exactly one product that satisfies all hard constraints and is optimal on all comparative preferences.

Taxonomy of eight steering mechanism types

The paper distills 8 types of mechanisms from regulatory and academic literature, each with an independent toggle:

1. Sponsored placement: Paid or promoted products are placed at the top of search results and labeled as sponsored.
2. Preferential ranking: The default ranking prioritizes products favored by the platform.
3. Drip pricing: The list page displays only the base configuration price; the full price requires visiting the detail page.
4. Promotional framing: Discount labels and genuine reference original prices are attached to specific products.
5. Defaults & bundling: Pre-selected add-on items or upgraded versions are included by default.
6. Scarcity & social proof: Displays low-stock or high-view count cues.
7. Trust signals: Highlights platform badges and genuine positive reviews.
8. Friction & obstruction: Adds extra steps to actions not favored by the platform.

The set of products favored by steering mechanisms is randomly chosen and strictly excludes the user-optimal product, ensuring all information is fully truthful and verifiable.

Mitigation strategies: CAVEAT-Harness and CAVEAT-27B

• CAVEAT-Harness: Builds on BrowserUse by adding a structured task specification component (TaskContract, which solidifies user hard constraints and preferences) and a decision verification tool (decision_checkpoint, which checks search coverage and unresolved evidence before checkout).
• CAVEAT-27B: Post-trains Qwen3.5-27B in two stages. The first stage performs supervised fine-tuning using trajectories from a teacher model (GPT-5.6-Sol-low) in a synthetic shopping environment; the second stage uses DAgger for online policy correction training on notebook tasks.

The paper conducted comprehensive evaluation and mechanism diagnostics across multiple benchmark subsets and model configurations.

Experimental Setup

• Models tested: The main evaluation covers 5 model families: GPT-5.6-Sol-low, Kimi-K2.6, Grok-4.3, Qwen3.5-122B-A10B, DeepSeek-V4-Flash-low; capability extension evaluations include GPT-4o, GPT-4.1, GPT-5 (nano/mini/main), GPT-5.5 (low/medium/high), GPT-5.6 (Luna/Terra/Sol series), the specialized embodied model Fara1.5-27B, and open-source fine-tuned models Qwen3.5-27B and CAVEAT-27B.
• Dataset and scale: CAVEAT-Standard has 52 tasks in total (each task has about 70 items, including 20 CAVEAT-Shop tasks); CAVEAT-Hard includes 5 CAVEAT-Shop tasks (each task has 2,112 items, displayed across 88 pages).
• Baselines and scaffolds: The default uses BrowserUse v0.13.6 native system prompts; comparison groups include BrowserUse + Prompting and CAVEAT-Harness; Fara1.5-27B uses its native Fara scaffold. Each step inputs a page screenshot and interactive element text, with a step limit of 250 steps (no truncation occurred in the experiments).
• Metrics and evaluation: The primary metric is the Optimal Purchase Rate (%), determined by programmatic comparison of checkout order records with ground-truth labels, without a model judge; supplementary metrics include the Graded Preference Score (P* ∈ [0, 1]) and the Promoted-Purchase Rate (%).
• Number of runs: Each task in CAVEAT-Standard is repeated 3 times; the main test in CAVEAT-Hard is repeated 10 times per task.

Main Results

On all 52 tasks in CAVEAT-Standard, the average and per-model performance of the 5 model families is as follows (according to Figure 3 and Table App.3):

Swipe horizontally to read the table

• Significant degradation across the board: The authors note that after enabling the steering mechanisms, the optimal purchase rates of all 5 models dropped by 37.2 to 78.9 percentage points.
• Systematic flow to promoted items: In the failed rounds of the steering group, about three-quarters shifted to 6 specific platform-recommended items, with a total purchase rate of 62.8%, compared to only 1.3% in the control group.
• Extreme degradation on large catalogs: On CAVEAT-Hard with 2,112 items, GPT-5.6-Sol-high had an optimal purchase rate of 90.0% in the control group, but failed in all 50 tests in the steering group, dropping to 0.0% (Figure 3C).

Single-Mechanism Ablation Experiments

On the 20 CAVEAT-Shop tasks, each mechanism was enabled individually for GPT-5.6-Sol-low (60 runs per condition, Table App.2):

• Visibility control mechanisms have the largest impact: Enabling only sponsored slots reduced the optimal purchase rate to 63.3% (down 36.7 pp); enabling only priority ranking reduced it to 75.0% (down 25.0 pp); enabling only split pricing reduced it to 83.3% (down 16.7 pp).
• Marginal mechanisms have smaller impacts: Default bundling dropped to 90.0%, promotional framing to 93.3%, friction hindrance to 95.0%, trust signals to 96.7%, and urgency to 98.3%. Enabling all eight dropped it to 51.7% (down 48.3 pp).
• Authors' interpretation: The sum of the individual mechanism reductions is 105 pp, while the all-on reduction is far below simple addition; the authors state this indicates overlapping effects among mechanisms, which act primarily through common decision vulnerabilities.

Failure Mode Diagnostic Analysis

The authors sampled 200 trajectories for analysis and designed controlled validations, identifying three major failure modes:

• Premature Search Closure: When a suboptimal item was moved to a sponsored slot, its visit rate rose from 45/60 to 60/60, and its purchase rate jumped from 0/60 to 48/60; in CAVEAT-Hard, all 50 failures of GPT-5.6-Sol-high stayed on page 1 (of 88 pages).
• Objective Drift: In CAVEAT-Hard, when the order of two equally weighted preference statements was swapped, 45/50 purchased the item optimal for preference A when A was first, and 41/50 purchased the item optimal for preference B when B was first; the agent claimed in its reasoning that the user prioritized the first preference.
• Premature Commitment with Unresolved Evidence: Analysis found 23 instances where the agent had browsed the optimal item but abandoned it due to misreading or forgetting attributes; with split pricing enabled alone, GPT-5.6-Terra-low produced 35 suboptimal purchases in 60 runs, all of which purchased split-priced items.

Intervention and Mitigation Experiments

Interventions were tested on CAVEAT-Shop and CAVEAT-Hard (Figure 5):

• CAVEAT-Harness effect: On CAVEAT-Shop, it improved GPT-5.6-Terra-low from 11.7% to 66.7% (the prompting baseline only reached 30.0%); on CAVEAT-Hard, it improved GPT-5.6-Sol-high from 0.0% to 80.0% (the prompting baseline only reached 13.3%). In the CAVEAT-Hard ablation, the verification tool alone contributed 66.7%, and the task specification alone contributed 6.7%.
• Post-trained CAVEAT-27B effect: On 16 CAVEAT-Shop tasks with held-out item categories, the native Qwen3.5-27B only reached 4.2% when combined with CAVEAT-Harness, while the post-trained CAVEAT-27B reached 22.9%; on 4 CAVEAT-Stay cross-environment tasks, it reached 16.7% (native was 0.0%).

Limitations and future directions noted by the authors

• Stress-test nature of concurrent mechanisms: The main experiment activates all 8 steering mechanisms simultaneously, constituting a controlled stress test rather than an estimate of the actual failure frequency of agents on deployed platforms (Section 8).
• In-depth analysis and interventions concentrated on a single marketplace: Deep analysis, scaffolding validation, and model post-training focus primarily on CAVEAT-Shop; cross-environment transfer was only examined on a small scale in CAVEAT-Stay (Section 8).
• Some diagnostic experiments have small sample sizes: Some targeted analyses rely on experimental samples of tens of rounds (Section 8).
• Boundaries of environment and task settings: The benchmark studies markets with clear goals, a unique optimal solution, and fixed mechanisms, and does not yet extend to other domains such as financial services, recruiting, or information platforms (Section 8).
• Dynamic preferences and adaptive gaming: Scenarios where users have partial or evolving preferences are not covered, nor are situations where platforms adapt over time in response to agents (Section 8).
• Overhead of scaffolding interactions: Reducing the additional interaction cost introduced by CAVEAT-Harness is an important future direction (Section 8).

Experimental coverage

• Environments tested: Includes 9 self-built simulated marketplaces; main in-depth ablation and mitigation experiments focus on CAVEAT-Shop, with transfer validation on 4 tasks in CAVEAT-Stay.
• Tasks and scale: CAVEAT-Standard evaluation covers 52 tasks, each with approximately 70 items, repeated 3 times; CAVEAT-Hard covers 5 tasks, each with 2,112 items, repeated 10 times.
• Steering mechanisms: Encompasses 8 types of commercial steering mechanisms, all implemented in a non-deceptive form (product attributes and prices are fully truthful), excluding purely deceptive means such as fake reviews or hidden fees.
• Model coverage: 18 model configurations were evaluated, with main reasoning models including GPT series, DeepSeek-V4-Flash, Kimi-K2.6, Qwen3.5-122B, Grok-4.3, and Fara1.5-27B.
• Costs and steps: The experiment set a 250-step limit and no step truncation occurred; CAVEAT-Harness increased runtime by 2.6 times and operational steps by 2.3 times compared to the baseline.

• Core positioning and problem: The paper studies whether computer-using agents can adhere to user goals under truthful information when facing online environments with self-interested preferences (such as commercial promotion and ranking bias), and proposes the first systematic benchmark, CAVEAT.
• Mechanisms and benchmark construction: It constructs high-fidelity web environments covering 9 real e-commerce and service domains, distills 8 types of commercial steering mechanisms such as sponsored slots, priority ranking, and split pricing, and uses programmatic verification to ensure that each task has a unique, verifiable user-optimal solution.
• Core experimental findings: On 52 standard tasks, the optimal purchase rate of 5 mainstream model families dropped from 78.6% in the control group to 17.3% in the steering group; on the large-catalog CAVEAT-Hard, GPT-5.6-Sol-high dropped sharply from 90.0% in the control group to 0.0%, and 62.8% of failed purchases went to specific platform-recommended items.
• Three failure mechanisms: Through trajectory and ablation diagnostics, three decision vulnerabilities were identified: premature search termination relying on the initially displayed results, objective drift influenced by irrelevant cues such as order, and premature commitment before key evidence such as price is resolved.
• Effective mitigation: A structured task constraint and inference-time evidence verification scaffold, CAVEAT-Harness, was proposed, restoring GPT-5.6-Sol's optimal purchase rate to 80.0% on the complex large catalog; the open-source model CAVEAT-27B combined with on-policy post-training also achieved significant improvement.
• Authors' conclusion and implications: Evaluating entrusted agents requires not only examining their task completion ability in neutral environments, but also testing their goal robustness when the environment has its own self-interested demands.