MemVenom: Trigger-Based Poisoning Attacks on Multimodal Memory of Web Agents

Core question: how to inject image-text malicious evidence into the graph-structured multimodal external memory of memory-augmented Web agents, so that it is persistently recalled and manipulates downstream actions.

• Problem setting: modern Web agents rely on external memory to store past observations and retrieve relevant experience; unlike transient prompt input, once a memory entry is written it can be repeatedly retrieved and continuously influence future decisions, which the authors consider a new security boundary.
• Shortcomings of existing methods: the authors state that existing agent security research mostly focuses on prompt injection, jailbreak, or pure-text retrieval poisoning, acting on the current input context or assuming pure-text memory; memory poisoning in the multimodal Web agent setting remains a gap.
• Core observation: the authors' key insight is that an effective attack must simultaneously optimize the probability that the malicious memory is retrieved and the degree to which it manipulates downstream decisions after retrieval.
• Proposed in the paper: MemVenom, a unified black-box two-stage attack framework, evaluated on 3 Web agent frameworks and 4 VLMs.
• Threat model (Section 3.1):
  • Goal: induce an attacker-specified side goal gm while maintaining normal utility under benign observations.
  • Knowledge: realistic black-box, with no access to the proprietary VLM, retriever internals, or memory scoring function.
  • Capability: memory-level access only, allowing injection of a small number of multimodal memory entries (text, images, graph relations) before deployment or before offline indexing.
  • Victim system: a multimodal Web agent using graph-structured external memory (Qdrant vector retrieval + Kuzu relational storage).

Memory-augmented Web agents

• Yao et al. (2023), Zheng et al. (2024), He et al. (2024), Deng et al. (2023), Koh et al. (2024)——enable Web agents to operate real websites, understand multimodal observations and execute multi-step tasks.
• Edge et al. (2024), Guo et al. (2024, 2025), Hu et al. (2024)——introduce external memory, retrieval modules or retrieval-augmented knowledge bases to store past observations and task experience. The paper notes that retrieved memories enter the policy input and become an active part of the observation–reasoning–action loop.

Web agent red-teaming and attacks

• Zhan et al. (2024), Debenedetti et al. (2024), Zhang et al. (2025b), Evtimov et al. (2025), Xiang et al. (2024a), Jia et al. (2025)——study prompt injection, jailbreak, malicious tool use, environment manipulation and agent backdoors. The paper notes that these attacks mostly act on the current input context, the immediate environment or the backbone model itself, and their effects are often tied to a specific interaction instance.

Retrieval and memory poisoning

• Zhong et al. (2023), Zou et al. (2025), Chen et al. (2024), Dong et al. (2025), Srivastava and He (2025), Jing et al. (2026)——show that RAG systems and memory-based agents are susceptible to corpus poisoning, backdoor triggers and malicious memory injection.
• Liang et al. (2025), Xu et al. (2026), Sunil et al. (2026)——make the retriever return adversarial documents or guide the language model to generate text responses desired by the attacker. The paper notes that these works are mostly pure-text settings, whereas the retrieval queries of Web agents are multimodal observations such as screenshots, and the attack effect must manifest in executable Web actions.

Comparison baselines and benchmark: Table 3 compares MemVenom with AgentPoison, CPA and BadChain adapted to this scenario; the task set comes from SeeAct (Mind2Web-style protocol, 10 categories of real websites).

The authors position this paper as extending retrieval and memory poisoning from pure-text RAG to multimodal memory-augmented Web agents, requiring joint optimization of malicious recall and action inducement after recall.

Overall approach: MemVenom first optimizes a visual trigger so that trigger-bearing webpage screenshots recall the malicious memory with high probability, then constructs a memory prioritization component unrelated to the specific malicious goal, and finally assembles it with a replaceable target component into a malicious subgraph injected into clean graph memory.

Problem formulation

• The agent receives a user goal gu, observes the webpage state ot at each step (mainly screenshots), retrieves rt=R(ot,M) from graph memory M=(V,E), and generates action at=π(gu,ot,rt).
• Memory nodes can contain image-text evidence, and edges encode semantic relations; retrieval returns directly matched nodes and graph-connected context.

Stage one: retrieval-stage optimization

• Collect a representative set of clean webpage screenshots Qraw; a rendering operator Tτ places the trigger τ in a fixed region of the screenshot, yielding the trigger-bearing screenshot set Qτ.
• Optimization objective τ^⋆=argmin L_cmp(τ)−λ_sepd(C(τ),C_raw)+λ_regL_reg(τ)+λ_spaceL_space(τ): make trigger-bearing observations form a compact cluster in the retrieval embedding space while separating them from clean observations and the benign memory space; Ω constrains the trigger's position and size.
• The screenshot x*ret whose embedding is closest to the trigger cluster center is taken as image evidence, and a recall-oriented memory node vret is constructed.

Stage two: post-recall composite visual attack

• This component does not encode a specific malicious goal, only a goal-agnostic memory prioritization signal, so it can be reused when changing goals.
• An ℓ∞-bounded perturbation is optimized on surrogate encoders fe, ft (the victim model is black-box); a bridge image is introduced as an intermediate visual anchor between the base image and the prioritization instruction, and updated iteratively.
• Base image objective L_base=−λ_imgcos(f_e(x′),f_e(B_pri⁽ⁱ⁾))−λ_textcos(f_e(x′),f_t(p_k^⋆))−λ_locmeancos(f_e^loc(x′),f_e^loc(B_pri⁽ⁱ⁾)), aligning the perturbed image with the bridge image and the rewrite set Ppri of the prioritization instruction.
• Finally, covert OCR injection embeds the fixed prioritization instruction τpri into the perturbed image (Algorithm 2): text is written at candidate positions with adaptive contrast, subject to an OCR budget ϵocr.

Malicious subgraph assembly and goal modularity

• The malicious subgraph Vadv(gm)=Vret∪Vgoal(gm)∪Vpri, with edges Eadv connecting the recall cue, malicious goal content, and prioritization evidence into a coherent subgraph, is injected into clean memory to obtain Mpoison.
• Changing to a new side goal only requires replacing Vgoal(gm); the recall component and prioritization component remain unchanged, with no need to re-optimize the trigger or visual attack.

Success criteria

• ASR-r=P(R): the probability that at least one injected malicious memory is retrieved; ASR-a=P(A|R): the conditional probability of executing an action consistent with the side goal after retrieval; ASR-ra=P(R∩A): the end-to-end success rate, ASR-a=ASR-ra/ASR-r.
• BU/PU measure benign task utility before and after poisoning, using Qwen3-VL-PLUS as an LLM judge to determine whether the original benign goal is completed.

Experimental setup

• Victim agents: SeeAct, LiteWebAgent, ReAct-style Web agents, all equipped with the same external memory recall interface.
• VLM backbones: Qwen3-VL-4B-Instruct, Qwen3-VL-8B-Instruct, Qwen3-VL-PLUS, GPT-5.4, covering open-source and API systems.
• Tasks: the task set released by SeeAct (Mind2Web-style protocol, 10 categories of real websites); each benign task is paired with an attacker-specified side goal.
• Side goals: four categories of OWASP-style task substitution goals—Phishing/Redirection, Controlled Privacy Leakage, Unauthorized Financial Operation, Destructive Data Operation—all routed to localhost sandbox pages, with deterministic success checkers (Table 6).
• Metrics: ASR-r, ASR-a, ASR-ra, BU, PU; benign utility is judged by Qwen3-VL-PLUS as an LLM judge.
• Memory implementation: Qdrant for image-text vector retrieval, Kuzu for semantic relation storage. The paper does not report the number of tasks or the number of repetitions.

Main results

Table 1 (Phishing/Redirection, ASR-r/ASR-a/ASR-ra, %):

Swipe horizontally to read the table

Backbone VLM	SeeAct	LiteWebAgent	ReAct
Qwen3-VL-4B-Instruct	98.04/46.59/45.68	96.05/89.77/86.22	95.59/76.31/72.94
Qwen3-VL-8B-Instruct	96.23/97.65/93.97	97.01/87.50/84.88	97.32/82.35/80.15
Qwen3-VL-PLUS	99.15/90.20/89.34	96.49/64.71/62.44	94.84/97.12/89.26
GPT-5.4	90.67/88.24/80.00	98.00/94.12/92.24	99.15/100.0/99.15

• The authors state that the consistently high ASR-r indicates that observations with the trigger can reliably recall the malicious memory, and that ASR-a and ASR-ra indicate that the recalled evidence can further induce the attacker-specified action.
• Exceptions: GPT-5.4 + ReAct on Controlled Privacy Leakage has ASR-r dropping to 53.57 and ASR-ra only 22.52 (Table 1c); GPT-5.4 + SeeAct on Unauthorized Financial Operation has ASR-a of 29.41 (Table 1b). The authors do not explain these lower combinations.
• Benign utility (Table 2): in most settings PU is close to BU, and the authors state that the attack is not achieved by degrading agent capability; however, on SeeAct the PU for Phishing/Redirection is 11.76, lower than the BU of 17.65, while on LiteWebAgent the PU is 41.18, higher than the BU of 23.53.

Comparison with adapted baselines

• Table 3 (Qwen3-VL-8B + SeeAct): MemVenom's ASR-ra is 93.97, AgentPoison is 74.28, CPA is 19.46, BadChain is 22.23; on Qwen3-VL-PLUS + ReAct, MemVenom is 89.26 and AgentPoison is 86.91.
• The authors state that AgentPoison's malicious recall is competitive in some cases but its ASR-ra lags behind, indicating that retrieval-oriented poisoning alone is insufficient to reliably manipulate Web actions.

Ablation study (Table 4)

• Composite visual attack: pure-text poisoning has ASR-ra of 73.59, adding covert OCR raises it to 84.91, and adding adversarial perturbation reaches 93.97 (the setting does not specify the backbone in the table caption; the values are the same as the Qwen3-VL-8B + SeeAct row in Table 1).
• Two-stage structure: after removing the recall-oriented component, ASR-r is 0.00 and ASR-ra is 0.00; after removing the prioritization component, ASR-ra drops from 93.97 to 75.18. The authors state that both stages are necessary.

Defense evaluation (Table 5)

• Two plug-and-play defenses were tested: Hardcoded Safety Rules and LlamaGuard. On SeeAct, ASR-ra drops from 93.97 without defense to 77.65 and 75.89 respectively; on ReAct, LlamaGuard reduces ASR-ra from 80.15 to 50.71 (RR 29.44).
• The authors state that both defenses only partially reduce the end-to-end success rate, and that existing lightweight defenses are insufficient against multimodal memory poisoning.

Other ablations and analyses

• Retriever transfer (Figure 3): the authors state that the diagonal performs strongly and off-diagonal transfer exists to some extent, and that the trigger is not merely overfitted to a single retriever; the converted text does not give specific values.
• Poisoning budget (Figure 5, Appendix A): the authors state that a small number of poisoned recall nodes can achieve higher malicious recall than adapted CPA and random poisoning, and that random insertion is almost ineffective.
• Top-k recall (Figure 4, Appendix A): the authors state that with trigger-bearing observations ASR-r remains high as k varies, and without the trigger malicious recall remains low.
• Trigger optimization process (Figure 14, Appendix G): in the projected embedding, red trigger-bearing query points move away from gray benign screenshot points and green benign memory points over iterations, forming a separated cluster.

Limitations and future directions noted by the authors

• Evaluation environment (Limitations): the evaluation is conducted in a controlled sandbox environment and selected Web agent frameworks; real deployments may involve more heterogeneous memory pipelines, retrieval strategies, and action constraints.
• Memory types (Limitations): the attack focuses on graph-structured multimodal memory for screenshot retrieval; tool-specific memory, user-profile memory, and dynamically updated episodic memory may have different security properties.
• Defense coverage (Limitations): the defense evaluation covers only lightweight plug-and-play protections; defenses that can jointly reason about memory provenance, retrieval consistency, task alignment, and action consequences remain to be explored, and the authors leave memory-aware defenses to future work.

Scope of experimental coverage

• The evaluated systems are 3 Web agent frameworks (SeeAct, LiteWebAgent, ReAct-style) × 4 VLM backbones (Qwen3-VL-4B/8B-Instruct, Qwen3-VL-PLUS, GPT-5.4).
• Side goals cover 4 categories of OWASP-style task substitution goals, all executed on localhost sandbox pages, with deterministic success checkers (Table 6, Appendix E).
• The defense evaluation includes two: Hardcoded Safety Rules and LlamaGuard (Table 5).
• Benign utility judging uses Qwen3-VL-PLUS as an LLM judge (Appendix D), and the paper does not report agreement between the judge and humans.
• The paper does not report the number of tasks per condition, the number of repetitions, the iteration budget for trigger optimization, or the query overhead.

An attack paper targeting memory-augmented Web agents, studying multimodal memory poisoning, an attack surface the authors consider overlooked.

• Problem: external memory is persistent and can be repeatedly retrieved, so injected malicious image-text evidence can influence agent decisions long-term; existing attacks mostly target prompts or pure-text memory.
• Method: MemVenom is a black-box two-stage framework—first optimizing a visual trigger so that trigger-bearing screenshots form a separated cluster in the retrieval embedding space and recall malicious nodes with high probability; then using adversarial perturbations optimized on surrogate encoders plus covert OCR injection to construct a target-agnostic memory prioritization component; the two are assembled with a replaceable target component into a malicious subgraph injected into graph memory, and changing the side goal only requires replacing the target component.
• Main results: GPT-5.4 + ReAct reaches ASR-ra of 99.15% on Phishing/Redirection (Table 1); on Qwen3-VL-8B + SeeAct it is 93.97, higher than the adapted AgentPoison's 74.28 (Table 3).
• Exception: GPT-5.4 + ReAct on Controlled Privacy Leakage has ASR-ra of only 22.52% (Table 1c), which the authors do not explain.
• Ablation and defense: after removing the recall component, ASR-ra drops to 0.00 (Table 4); LlamaGuard reduces ASR-ra on ReAct from 80.15 to 50.71, failing to eliminate the threat (Table 5).
• Authors' conclusion: external memory is a security-critical component in Web agent systems and requires memory-aware defenses capable of reasoning about memory provenance, retrieval behavior, and task alignment.