Visual Confusion Agents: Exploiting and Defending Against Perception Failures in Computer-Use Agents

CUA may authorize clicks based on a misread screen, thereby executing actions that would not be authorized under correct perception.

• Occurrence scenario: CUA issues click(x,y) on a screenshot; coordinates have no semantics outside the screenshot. The authors argue that perception is the trust anchor for what action is being authorized, not an ordinary input. The introduction cites OSWorld-MCP: 56.7% of actions in 369 tasks did not hit the intended element.
• Existing shortcomings: The authors state that existing work mostly asks whether the task is completed, rarely whether the click landed on the correct object. Access control on tool names and parameters cannot read coordinates; in-agent guardrails and LLM self-checks share the same perception and can be bypassed when the runtime is compromised.
• Three sources: non-adversarial visual localization errors, screenshot tampering before the model sees it, and TOCTOU between screenshot() and click(). The introduction cites ScreenSpot-Pro accuracies of 0.8% for GPT-4o, 18.9% for OS-Atlas-7B, and 43.6% for Qwen2.5-VL-72B.
• This paper proposes: The authors claim to be the first to formalize this as a visual confused deputy, and demonstrate with ScreenSwap that pixel swapping can redirect clicks to high-privilege buttons, then propose out-of-process dual-channel contrastive classification.
• Threat model:
  • Deviation definition: An action based on perceived screenshot s′ lands on the real element e_actual, while the model believes it is e_perceived, which differs (Definition 1).
  • Position and capability: The runtime sits between the screenshot and the click; the authors call it a structural man-in-the-middle. It can modify screenshots via supply chain or tool poisoning, and the screen may change between two calls.
  • Victim: CUA operating graphical interfaces via low-level GUI commands.
  • Enforcement point: The guardrail intercepts clicks, screenshots, and reasoning text out-of-process; the runtime is untrusted, and the classifier is the only enforcement point (§5).

Section 7 of the paper divides related work into benchmarks that record failures, authorization for tool calls, and verification that separates generation from authorization.

CUA benchmarks

• OSWorld-MCP (Jia et al., 2025) — evaluates tool calls in desktop multi-step tasks and reports that actions often miss the target.
• ScreenSpot-Pro (Li et al., 2025b) — examines visual localization on professional GUIs in isolation. Appendix G compiles published accuracies on this benchmark, not the authors' own model experiments.
• OS-Harm, CUAHarm, AgentHarm (Kuntz et al., 2025; Zhang et al., 2025; Andriushchenko et al., 2024) — link failures to specific harms or policy-violating actions. The authors state that these five benchmarks document the problem but none provide a structural defense outside the perception loop.

Agent safety

• Wang et al. (2025) — systematically catalogs CUA-specific vulnerability classes. This paper's TOCTOU source cites that work.
• CSAgent (Park et al., 2025) and Delegated Authorization (Gupta and Singh, 2025) — authorize within the agent or by task scope. The authors argue such mechanisms are insufficient for click(x,y) that has no semantics outside the screenshot.
• ToolSafe (Zhang et al., 2026) and AgentGuard (Kim et al., 2025) — study semantic tool names and parameters in API-based agents. §2 also lists AgentSentinel (Liu et al., 2025) as an in-agent guardrail that can be bypassed by a compromised runtime and makes the same localization errors.

External verification and confused deputy

• Self-checking and accompanying authorization (OpenAI, 2025; Anthropic, 2024; and the above Park, Gupta, Kim works) — judge based on the model's own representation of the task, tool call, or screen. The authors argue that when perception is already wrong, self-checking is circular.
• Confused deputy (Hardy, 1988) — a program misuses privileges because designation and authority are not separated; capability systems (ERights.org, 2000) handle this by aligning the two. The authors extend this perspective to screen agents that infer designation from pixels.

Baselines and data: The main experiments compare against image channel, text channel, OR, AND, and image+text veto, as well as mean/max aggregation; Table 3 also compares raw SigLIP (768-dim) with the authors' 384-dim projection. Data include ScreenSpot-Pro, OS-Harm, a self-built SOC panel, neutral button scenarios, and VLM-Reasoning-Traces. The paper does not use another published CUA guardrail as a main result baseline.

The authors position this paper as: moving verification out of the agent's perception loop, independently classifying visual click targets and text rationales before execution.

Before executing a click at runtime, dual-channel contrastive classification uses a deployment-side knowledge base to classify the click crop and the reasoning text separately, and if either path is judged restricted, it blocks.

Problem Setting and ScreenSwap

• The agent loop is screenshot(), handing pixels to the LLM, then issuing click(x,y). Definition 1 writes the vulnerability as a=f_LLM(s′), e_actual≠ e_perceived: the action is generated from the perceived screenshot, but the coordinates land on another real element.
• The three causes are independent of each other: localization errors, adversarial modification of the screenshot, and TOCTOU. The authors argue the boundary between ordinary misclicks and exploitation is small.
• ScreenSwap: a compromised runtime swaps the pixel regions of two buttons before the screenshot is sent to the LLM (Figure 1). The example is a normal confirm button and a restricted credential reset button on an SOC panel. The authors say the image code is 8 lines, button localization adds about another 30 lines, and it uses real rendered pixels. Appendix A gives the process and code.
• Properties the authors list: the LLM does not see the real screen, coordinate policies cannot see semantics, if only the model-observed screenshot is logged the audit trail looks consistent after the fact, and it is hard to distinguish from the existing 56.7% misclick rate.

Dual-Channel Classification

• The guardrail intercepts each click outside the agent process, with input being the screenshot, coordinates, and reasoning text t (Figure 2). The classifier is multi-modal-embed-small: 120M parameters, SigLIP image encoder, MiniLM text encoder, trained 384-dimensional projection, shared embedding space.
• Image channel: crops around the click, specified as 100×100 in Appendix D, then after SigLIP does nearest-neighbor cosine similarity classification against the restricted library Kimg− and the allowed library Kimg+.
• Text channel: encodes t with MiniLM, comparing against Ktxt− and Ktxt+.
• The paper does not give a single numerical threshold unified across all experiments; the SOC experiment reports the interval |Δ| of the difference between the two library scores.

Fusion Rules

• OR: if either path is marked restricted, block; otherwise allow. The authors' rationale in §4 is that the two paths correspond to different failure surfaces; AND would miss threats with neutral appearance.
• image+text veto: the image channel makes the decision, and the text channel only adds blocking signals. Appendix E suggests using veto for localization detection and OR for the neutral button flow. The numbers for the two rules are written separately in question 4; this article does not treat them as the same decision.

Knowledge Base and Aggregation

• The text library baseline is 10 restricted descriptions plus 10 allowed descriptions, with ablations expandable to 20+20; Appendix D gives the full text, not listed here.
• The image library is dynamically generated from GTK3/Ubuntu-style synthetic button crops, including light and dark, as well as danger styles, success styles, and neutral styles (Figure 4).
• Aggregation compares mean (centroid) and max (nearest neighbor). The authors say max is equivalent to the zero-temperature limit of contrastive learning softmax and consider it more appropriate for this inference-time guardrail (Appendix C).

Coverage Boundaries Written by the Authors

• §5: this classifier targets Cause 1 and partially Cause 2, which changes the element at the click coordinates. The image channel does not look at LLM intent, and the text channel does not look at control appearance. The residual risk is both paths missing simultaneously, denoted ε_cls, and it varies with deployment.
• The same section states it does not guarantee: harms in typed content, chains of individually allowed clicks, adversarial inputs against cosine similarity (not modeled), and interface patterns and intent descriptions outside the knowledge base vocabulary.

实验设置

• 分类器：§6的全部实验使用120M的multi-modal-embed-small（SigLIP+MiniLM+384维投影），以最近邻最大余弦相似度对知识库分类。论文未写统一数值阈值，也未报告重复次数或统计检验。
• SOC / ScreenSwap：GTK3面板跑在Xvfb 1024×768上，用scrot截图、xdotool点击；8个受控场景（附录B）。概念验证不含LLM推理调用。
• OS-Harm：13个场景，使用基准原文任务指令和Ubuntu风格界面（§6.2）。留一法另用99条有害指令，以及从374条良性指令中随机抽的100条（§6.5）。
• ScreenSpot-Pro：260张真实截图，26个专业应用、三个平台；每样本1次正确点击加3次定位错误，每种配置1040例（Table 1、§6.3）。
• 中性按钮与轨迹：20个场景、5类中性按钮，危险与安全各10（§6.4、附录F）。VLM-Reasoning-Traces有501条Claude 3.7 Sonnet扩展思考轨迹；随机子集为200、100条安全加15条合成危险推理、50（§6.6）。作者没有在本文中运行该Claude模型。
• 指标与对照：Recall、Specificity、F1、Accuracy，以及§5的ε_cls。对照为单通道、OR、AND、image+text veto、mean/max，以及Table 3的原始SigLIP（768维）。没有LLM评审。

主结果

Swipe horizontally to read the table

设置	Recall	Specificity	F1	Accuracy	出处
ScreenSpot-Pro，image+text veto	87.4%	88.8%	.915	87.8%	Table 1
ScreenSpot-Pro，image cross-app (max)	83.1%	88.8%	.889	84.5%	Table 1
ScreenSpot-Pro，dual agree	69.9%	71.5%	.779	70.3%	Table 1
中性按钮，image-only	0%	100%	.000	50%	Table 2
中性按钮，text-only	100%	100%	1.00	100%	Table 2
中性按钮，Dual (OR)	100%	100%	1.00	100%	Table 2
OS-Harm 13场景，Original+max	100%（7/7）	67%（4/6）	.875	84.6%	Table 5、附录C
OS-Harm留一法，max	96.0%	98.0%	0.969	未报告	§6.5

• 表注称image+text veto是最优融合。§6.3称该融合比仅图像多拦住34个定位错误且不增加假阳性；同节称文本通道在24.2%的正确点击上与真值不一致，并因此把dual agree拉低。Table 1里veto与image cross-app (max)的特异度同为88.8%。论文没有解释这24.2%为何没有变成veto的假阳性。
• 摘要称联合护栏始终优于任一单通道。Table 2中Dual (OR)与text-only持平；附录B的Table 4里，按钮为危险样式时dual OR与image-only同为8/8。AND在Table 2中与image-only同为F1 .000。作者用Figure 3说明：定位错误主要由图像通道覆盖，中性按钮上的危险意图主要由文本通道覆盖。
• OS-Harm的7次拦截中，作者称只有Software Updater上点到Install Now而非Cancel是意图与目标不符，其余6次是因为按钮本身呈危险样式。2个假阳性是合法Send和用户要求的Trash；作者称二者与有害用法在点击级外观相同（Table 6、附录C）。

ScreenSwap与受控面板

• 在GTK3/Xvfb上对调按钮区域后，点击管理按钮坐标会弹出凭据重置确认框。附录B.1写明该验证不含LLM调用，主张是对调后的坐标会打中真实管理按钮。
• 双通道拦住该管理目标，文中给出图像通道margin +0.46，且8个受控场景均被拦住（§6.1、附录B）。
• Table 3：Ours图像4/4，最小|Δ| 0.1779、平均0.3035；SigLIP图像4/4，最小0.1126、平均0.2088。作者称投影把最小间隔提高58%。文本2/4，跨模态3/4；作者称文本编码器训练于自然图像描述，不是UI描述。Table 4中，声称正常动作但坐标打在管理按钮上的两行，文本通道标为NORMAL，由图像通道拦住。

推理轨迹、留一法与跨域

• 200条轨迹上，指令与推理的配对排序准确率为96.5%，平均余弦间隔+0.26。
• 100条安全轨迹加15条合成危险推理：F1为1.000。把50条推理换成危险意图时，文本通道拦住50/50，正确推理放行46/50；作者称8%假阳性来自过短、泛泛的指令。
• 留一法改用mean后F1为0.627。4/99条有害指令被漏掉，作者称它们听起来像良性操作，危害在执行上下文而非措辞。InjecAgent知识库迁到OS-Harm测试时特异度为0%；作者称对比库必须含部署域中的代表性样本。

其他消融与分析

• Table 5：8种知识库配置的召回都是100%（7/7）。Original+max特异度67%、F1 .875；文本库扩到20+20并用mean时，特异度17%、F1 .737。作者称重叠条目会把良性动作拉向危险质心，聚合方式比库规模更要紧。
• Table 6标注Mount partition与Change Password由max改为放行。Table 5中Original+mean的真阴性是3，Original+max是4。
• 附录E：跨应用text-only（max）召回100%、特异度0%。朴素文本库加AND的准确率16.1%；按应用分组后dual-agree为70.3%，仍低于image-only的84.5%。
• 跨应用image-only（max）在26个应用中16个准确率不低于85%；文中最高为Photoshop 98%、DaVinci Resolve 95%、Origin 93%，最低为Premiere、Blender、PowerPoint，均为75%。
• 文本通道与真值不一致的正确点击占24.2%；文中写出的最高比例为Premiere 60%，DaVinci与Illustrator为50%（§6.3）。
• §5写出的ε_cls：受控SOC为0%，ScreenSpot-Pro的image+text veto为12.6%，中性按钮意图场景为0%，OS-Harm留一法为4.0%。§6.7：AMD MI300X上约14 ms/点击；作者称相对CUA推理的约200 ms到数十秒可忽略。

Limitations and Future Directions Noted by the Authors

• Single click (§8): Only looks at one click crop and one reasoning trace, so it cannot directly detect content-level harms that depend on document or interface state semantics, such as dangerous text in email bodies, terminal commands, or forms.
• Multi-step and task intent (§8): Does not reason over longer action sequences, so it cannot detect harmful plans composed of individually harmless clicks; nor does it handle cases where the agent correctly executes a harmful user request. The authors consider the latter to be task-level intent alignment, not interface misperception.
• Interaction types (§8): Better suited to pointer-driven GUIs; weaker on terminal interactions where risk is mainly in command semantics. The authors note that all 52 computer-use tasks in CUAHarm are terminal tasks.
• Channels and knowledge base (§8): The image channel still errs on small or visually ambiguous elements, with residual errors concentrated on low-salience targets like toolbar icons. The text channel degrades on heterogeneous cross-app data; the authors suggest enabling it only in pipelines where reasoning verification is informative. Both channels depend on a deployment-side knowledge base; curation imposes operational burden, and an incomplete or poorly designed base may affect robustness; the authors state that the image channel is more tolerant of such curation errors than the text channel, and point to Appendix C.
• Future directions (§9): The authors plan to move to sequence-level judgments, covering terminals, large text inputs, and document content semantics, and to reduce manual base construction via automatic induction, continuous updates, or adaptation from usage trajectories; they also mention combining with human confirmation for irreversible actions, or per-agent policies in shared or high-assurance environments.

Experimental Coverage

• The classifier is the 120M multi-modal-embed-small, with decisions via nearest-neighbor maximum cosine similarity; §6 does not report repetitions or statistical tests.
• Evaluations include 8 GTK3 SOC panel scenarios, 13 OS-Harm scenarios reproduced with original instructions, 260 ScreenSpot-Pro screenshots (1 correct click plus 3 localization errors per sample, 1040 cases), 20 neutral button scenarios, OS-Harm leave-one-out with 99 harmful plus 100 benign instructions, and three subsets of 501 trajectories from VLM-Reasoning-Traces: 200, 100+15, and 50 (§6).
• The ScreenSwap proof of concept performs real clicks on Xvfb and captures the confirmation dialog; Appendix B.1 states that it does not include LLM inference calls.
• Baselines include image/text single channels, OR, AND, image+text veto, and mean/max and knowledge base size (Tables 1, 2, 5, Appendices C, E). Cross-domain experiments apply the InjecAgent knowledge base to OS-Harm testing, with specificity 0% (§6.5).
• §5 states that attackers who can craft inputs to deceive embedding cosine similarity are not modeled; §6.7 reports approximately 14 ms/click on AMD MI300X. The paper does not report agreement with human judgments.

The authors frame CUA's perception failure as a security vulnerability, visual confused deputy: the agent authorizes click(x,y) based on a misread screen, and the actual landing point differs from the element it believes it is clicking.

• Problem: The runtime sits between the screenshot and the click. Deviations can come from localization errors, screenshot tampering, or TOCTOU. The authors argue that tool-name policies and in-agent self-checks cannot verify what is actually clicked.
• Exploitability: ScreenSwap swaps the pixel regions of two buttons before the screenshot is fed to the model. A GTK3 proof of concept triggered a credential reset confirmation dialog; this demonstration did not call an LLM (Appendix B.1). The authors state the image manipulation is about 8 lines.
• Guardrail: Out-of-process dual-channel contrastive classification uses SigLIP to compare click crops and MiniLM to compare reasoning, against a deployment-side allowed library and a restricted library. OR blocks if either channel flags risk; image+text veto lets the image decide and text only adds blocks.
• Results: On 1040 ScreenSpot-Pro cases, image+text veto achieves recall of 87.4% and F1 of .915 (Table 1). On neutral buttons, the image channel F1 is .000, while the text channel and OR are 1.00 (Table 2). On OS-Harm's 13 scenarios, it blocks 7/7 harmful clicks and allows 4/6 safe clicks. Leave-one-out max F1 is 0.969; when the InjecAgent knowledge base is transferred to OS-Harm, specificity is 0%.
• Authors' conclusion: The authors argue that CUA safety cannot only improve action generation; it must also verify, outside the perception loop, what the agent thinks it is clicking and why. They also state that the guardrail does not cover content-level harms, multi-step combinations, or terminal command semantics, and relies on a deployment-side knowledge base.