SPILLage: Oversharing by Web agents

Core question: when a web agent executes user tasks on live websites, does it disclose task-irrelevant user information (Sirrelevant) to third parties through observable action traces, which the authors call Natural Agentic Oversharing.

• Setting and importance: Users authorize resources such as email, calendars, and chat logs to web agents, with an implicit privacy expectation—the agent uses only the information needed to complete the task. Unlike chatbots, web agents act on the open web and leave action traces; every click, scroll, and navigation can be observed by websites.
• Gaps in existing work: The authors state that prior work (Zharmagambetov et al., 2025; Shao et al., 2025; Liao et al., 2025) studied "leakage" in adversarial settings (prompt injection, malicious websites) and treated only verbatim text leakage as a binary "detected or not" problem, ignoring behavioral channels, implicit disclosure, and degree characterization.
• Core observation: Even in non-adversarial settings, agents may overshare through four pathways: explicit/implicit text input, and explicit/implicit behavioral actions (Figure 1); text filtering can cover only one of these quadrants.
• Paper contributions: Propose the SPILL AGE framework (2×2 taxonomy), a benchmark of 180 tasks on Amazon and eBay, an LLM-Judge-based step-by-step auditing method, and report measurement results from 1,080 runs.
• Threat model (non-adversarial):
  • Goal: A passive observer infers task-irrelevant attributes Sirrelevant from the agent's action trace A.
  • Knowledge: The observer can only record observable actions and cannot see the user's original request or the agent's internal reasoning.
  • Capability: Strictly passive; cannot modify website content or manipulate the agent; website operators and analytics scripts can play the role of observer.
  • Victim system: Web agents using the Browser-Use or AutoGen framework, driven by gpt-4o/o3/o4-mini.

Web Agent

• WebArena, VisualWebArena, Mind2Web, etc. (Zhou et al., 2024; Koh et al., 2024; Deng et al., 2023; Gou et al., 2025) — build real or simulated web task environments to evaluate an agent's task completion ability. The paper points out that these benchmarks only measure task success rate and have no privacy annotations.
• Web agent surveys and systems (Yang et al., 2025a;b; Sapkota et al., 2026; Liang et al., 2023) — describe the ability of web agents to parse the DOM and operate interfaces in real dynamic web pages to complete tasks.

Privacy risks of web agents (grouped by paper)

• Adversarial information extraction: EIA (Liao et al., 2025) and WASP (Evtimov et al., 2025) study prompt injection embedded in web pages hijacking agents to leak data; Bagdasarian et al. (2024)'s context-hijacking redirects agent goals; Green et al. (2025) claim that reasoning chain traces can leak sensitive information. The paper points out that all of these assume the attacker modifies the web page or intercepts the model internals, which differs from the setting of this paper.
• Privacy knowledge–action gap: AgentDAM (Zharmagambetov et al., 2025) and PrivacyLens (Shao et al., 2025) claim that agents can correctly answer privacy questions yet leak in their actions. The paper points out that they evaluate in text-only environments or treat oversharing as a binary judgment, missing behavioral channels and the explicit/implicit distinction; these two are the closest works explicitly named by the paper.
• Platform-level and network-level leakage: Ukani et al. (2025) claim that agent frameworks may disable browser-level privacy protections; Jeong et al. (2026) claim that a passive network observer can infer user traits from the sequence and timing of visited domains. The paper states that the former is a framework configuration issue and the latter is network-layer metadata leakage, both orthogonal to the in-page actions of concern in this paper.

Contextual integrity

• Nissenbaum (2004; 2009) — proposed contextual integrity, judging whether information flows conform to contextual norms according to participants, information types, and transmission principles; this paper uses it as the theoretical basis for privacy expectations.

Baselines and comparison

• The paper has no baseline methods for comparison; Table 5 compares this paper with EIA, PrivacyLens, AgentDAM, WASP, DECEPTICON, Privacy Practice, and Network-Level on the four columns of threat model, channel, directness, and environment, and only SPILL AGE simultaneously covers Both/Both and uses the live web.

The authors position this paper as: prior work covers only one of content or behavior, only one of explicit or implicit, and mostly in simulated environments, whereas SPILL AGE simultaneously characterizes four types of oversharing on non-adversarial live websites.

Overall approach: translate the concept of "oversharing" from personal online behavior to web agents, formalize it into four categories along two dimensions—channel (content vs. behavior) and directness (explicit vs. implicit)—then construct tasks on real e-commerce sites that mix task-relevant and task-irrelevant attributes, and audit agent action trajectories step by step.

Problem setup and three-party model

• Three parties: User U provides prompt P (task request + resource access R, where R encodes the user's attribute set S); web agent W executes the task and produces an action trajectory A = {a1, ..., an}; passive observer O records A and attempts to infer Sirrelevant.
• Attribute partition: An attribute s is Srelevant if it is necessary to complete the task, otherwise Sirrelevant; the user expects the agent to rely only on Srelevant and not disclose Sirrelevant through any observable action.
• Two action types: Text input actions (Type) and behavioral navigation actions (Click, Scroll).

2×2 oversharing taxonomy

• Four categories: C = {CE, CI, BE, BI}, where C/B denote content/behavioral channel and subscripts E/I denote explicit/implicit (Figure 3). Explicit means Sirrelevant appears verbatim in the action; implicit means it can be inferred by an observer but is not stated.
• Oversharing event: Defined as a tuple (a, s, c), i.e., action a leaks attribute s through category c; the audit function F(a, Sirrelevant) maps each action to the set of detected events.
• Authors' rationale for the dimensions: They determine the available defenses (text filtering only covers content, string matching only covers explicit), determine who can observe (search box content is visible to the target site, navigation actions may be recorded by intermediate trackers), and allow diagnosing how agents fail.

Audit method

• Explicit (CE, BE): Detected using string matching.
• Implicit (CI, BI): The evaluator performs semantic reasoning about "what an observer can reasonably infer."
• LLM-Judge input and output: At each step, the evaluator (gpt-4o-mini) is given three inputs—the user's original prompt with Srelevant/Sirrelevant annotations, the executed action a, and the agent's stated next goal; it outputs structured JSON containing category c, involved attribute s, evidence, and reasoning (Figure 6 in Appendix B.4 gives the full evaluation prompt).

Benchmark construction

• Reasons for choosing live e-commerce (Amazon, eBay): Shopping tasks naturally mix Srelevant and Sirrelevant; the platforms provide both text-input and behavioral-navigation interaction surfaces; e-commerce sites record fine-grained user behavior and are realistic passive observers.
• Three-stage data generation: First define shopping tasks and generate 10 user attributes; then manually partition attributes into Srelevant/Sirrelevant; finally use claude-3.7-sonnet to render each persona into three prompt styles: chat, email, generic (examples in Appendix B.6).
• Quality control: Each generated prompt is human-verified to check that the attribute partition is correct, the task is completable on the target website, and the style is natural.
• Scale: 30 personas per style per site, totaling 90 tasks per site and 180 tasks across the two sites.

Metrics

• Occurrences (Occ.): The total number of oversharing events of a given category.
• Oversharing Rate (OR): The number of events divided by the total number of actions; can exceed 1.0 when a single action leaks multiple attributes.
• Task Success: For Browser-Use, determined by the backbone LLM via the success flag attached to the done action; for AutoGen, judged by an LLM (Figure 7, Appendix C.1).

Experimental setup

• Combinations tested: Frameworks Browser-Use and AutoGen (MultimodalWebSurfer), backbone models gpt-4o, o3, o4-mini, for 2×3 combinations in total.
• Benchmark: Self-constructed 180 tasks, 90 each on Amazon and eBay (30 personas each for the three styles chat/email/generic per site), with Srelevant/Sirrelevant human annotations.
• Run scale: 180 tasks × 2 frameworks × 3 models = 1,080 runs (about 10^5 API calls); each run has a 50-step limit and 5-minute timeout, with the browser session reset between tasks.
• Metrics: Occurrences, Oversharing Rate (OR, can exceed 1.0), Task Success; confidence intervals are 95% bootstrap CI (percentile method, 10,000 resamples).
• Evaluation method: Oversharing audits use gpt-4o-mini as LLM-Judge; AutoGen's task success uses an LLM judge, while Browser-Use uses its built-in completion detection.
• Main table coverage: Explicit results are reported in the main text for gpt-4o (Table 1, 2), with o3/o4-mini in Appendix Tables 9, 10; implicit results are reported only for Browser-Use (Table 3, 11).

Main results

Explicit oversharing, gpt-4o (per Table 1, 2):

Swipe horizontally to read the table

Site	Framework	Content Occ.	Content OR	Behavioral Occ.	Behavioral OR
Amazon	AutoGen	4	.013	223	.610
Amazon	Browser-Use	182	.117	905	.326
eBay	AutoGen	46	.018	342	.136
eBay	Browser-Use	58	.035	28	.016

• Behavioral types dominate: The authors report that on Amazon + Browser-Use the ratio of behavioral to content events is about 5× (905 vs. 182), and on eBay + AutoGen about 7× (342 vs. 46), and state that this pattern holds across all models and prompt types.
• Frameworks redistribute rather than eliminate risk: Browser-Use trajectories are long with many events; AutoGen has fewer steps but a higher per-step rate—under Amazon generic, AutoGen's behavioral OR reaches 1.027, which the authors state means a typical action leaks at least one Sirrelevant attribute.
• Prompt style moderates severity: The generic style consistently produces the highest oversharing rate (Amazon + AutoGen behavioral OR: generic 1.03 vs. chat 0.37).

Implicit oversharing

• Less frequent but non-zero: On Amazon, gpt-4o (Browser-Use) produces 325 implicit content and 45 implicit behavioral events (Table 3); eBay has 62 and 58.
• Model differences: The authors report that gpt-4o's implicit oversharing is an order of magnitude higher than o3's (Amazon content 325 vs. 12, behavioral 45 vs. 8, Table 3 vs. Table 11), and attribute this to stronger models inferring Sirrelevant-related concepts and propagating them across steps.
• Platform and style effects: Amazon's implicit content events are about 5× eBay's (325 vs. 62); the generic style has the highest implicit rate (Amazon content OR 0.171 vs. chat 0.127, email 0.046).

Defense and utility experiments

• Prompt-level mitigation (negative result): Adding an instruction to the system prompt to avoid using irrelevant information, tested with Browser-Use + gpt-4o on Amazon with 13 personas (chat style), raised the behavioral oversharing rate from 58.9% to 86.0%. The authors state that (preliminarily) emphasizing "task-irrelevant" may increase its salience in the model's reasoning, and warn about the risks of naive guardrails.
• Removing Sirrelevant before execution: After automatic removal with claude-3.7-sonnet and human verification, task success rate improved by up to 17.9% (Table 4), and Browser-Use overall accuracy rose from 73.4% to 99.4%; most cells in Table 4 reach 100.0%, with exceptions on eBay generic: Browser-Use 96.7% (-3.3), AutoGen 96.7% (+0.7).
• Qualitative study of commercial agents (Appendix D): Three commercial agents were manually monitored with 10 persona shopping prompts; the authors report that Brave AI Browsing and ChatGPT Atlas showed no detected oversharing, while Perplexity Comet repeatedly pasted large portions of user conversations into third-party search boxes, leaking sensitive personal information.

Other ablations and analyses

• Task success rate (Table 8): AutoGen overall 0.813 (gpt-4o) to 0.994 (o3), Browser-Use 0.742 (gpt-4o) to 0.929 (o3).
• o3/o4-mini explicit results (Table 9, 10): o3 has fewer behavioral events and more content events; o4-mini is in between, with 402 behavioral and 241 content events under Amazon + Browser-Use generic.
• Model behavior profiles (Table 6, Appendix A.2): The authors state that gpt-4o embeds multiple preferences into a single query, o3 writes preferences directly into actions, and o4-mini has the cleanest queries but generates a todo.md planning file that continuously records user intent.
• The paper does not report the agreement between LLM-Judge audit results and human judgments.

Limitations and future directions noted by the authors

• Only OpenAI models tested (Section 6.2): The evaluation focuses on OpenAI models; extending to other model families may exhibit different oversharing patterns; the authors supplement with a qualitative study of a commercial web agent.
• Tasks limited to the e-commerce domain (Section 6.2): The 180 tasks target e-commerce; the authors state that the taxonomy can generalize to domains such as healthcare, legal, travel, and finance where agents browse external websites on users' behalf.
• Single-website sessions only (Section 6.2): Agents are constrained to a single website, whereas production deployments often span multiple domains; cross-site action trajectories would allow third-party trackers to make richer behavioral inferences.
• Only the first defense direction explored (Section 6.1): The authors propose three directions—input-side sanitization, action-level monitoring, and model-aware guardrails—and this paper explores only input-side sanitization, leaving the latter two for future work.
• Causes of commercial agent behavior to be investigated (Appendix D): The authors state that future work should investigate the causes of Perplexity Comet's oversharing (information selection capability, utility priority, lack of guardrails, or architectural differences).

Experimental coverage

• The backbone models tested are gpt-4o, o3, and o4-mini, 3 in total; the frameworks are Browser-Use and AutoGen, 2 in total; the sites are Amazon and eBay (Section 5 Setup).
• Implicit oversharing is evaluated only on the Browser-Use framework (Table 3, 11); AutoGen's implicit results are not reported.
• The prompt-level mitigation experiment is conducted only with 13 personas, Browser-Use + gpt-4o, Amazon, and chat style (Section 5.4).
• The commercial agent study is manual monitoring of 10 prompts, with no public API (Appendix D).
• The paper does not report the agreement between LLM-Judge (gpt-4o-mini) audits and human judgments, nor the audit's accuracy or error rate.

An evaluation/benchmark paper on web agent privacy, focusing on the problem of agents unintentionally leaking task-irrelevant information while acting on users' behalf on real websites in non-adversarial settings.

• Problem: Users hold implicit privacy expectations when authorizing agents to access resources such as email and calendars; the authors state that prior work focused only on verbatim text leakage in adversarial scenarios, ignoring behavioral channels such as clicks and scrolling and implicit disclosure.
• Method: SPILL AGE divides oversharing into four categories by channel (content/behavior) × directness (explicit/implicit); constructs 180 tasks on Amazon and eBay with Srelevant/Sirrelevant human annotations (three styles: chat/email/generic); uses gpt-4o-mini as an LLM-Judge to audit action trajectories step by step, using string matching for explicit and semantic reasoning for implicit.
• Scale: Two frameworks, Browser-Use and AutoGen × three backbones, gpt-4o/o3/o4-mini, for a total of 1,080 runs.
• Main results: Oversharing occurs in all configurations; the authors report that behavioral events are about 5× content events (Amazon + Browser-Use + gpt-4o: 905 vs. 182); AutoGen has fewer steps but a higher per-step rate, with behavioral OR reaching 1.027 under Amazon generic; gpt-4o's implicit oversharing is an order of magnitude higher than o3's (Amazon content 325 vs. 12).
• Defenses and utility: Prompt-level mitigation instead raised the behavioral oversharing rate from 58.9% to 86.0% (preliminary experiment with 13 personas); removing task-irrelevant information before execution raised task success rate by up to 17.9%, and Browser-Use overall accuracy rose from 73.4% to 99.4%.
• Authors' conclusion: Protecting web agent privacy requires extending the definition of "output" from text to the agent's behavior on the web; the authors state that privacy and utility can be aligned, and plan to release the dataset and code.