MemSecBench: Tracing agent memory poisoning from persistence to consequences and repair

After malicious content is written into an agent's memory, it may persist across sessions, be retrieved, and cause verifiable external consequences, and it may not be fixable while preserving benign memories.

• Scenario: The authors argue that cross-session memory can carry preferences, project context, and task experience, supporting long-horizon tasks, and also allowing malicious content from one interaction to influence behavior after the session ends.
• Existing gaps: The authors state that existing evaluations mostly cover only parts of the lifecycle, lacking tracking of the same malicious semantics across persistence, verified consequences, and selective repair under fixed agents, models, and environments, and lacking paired comparisons of memory backends.
• Questions to answer: The same model and input may yield different outcomes depending on storage, retrieval, and update methods. The authors want to see whether malicious semantics persist after being written, whether they are retrieved, adopted, and turned into case-defined external consequences, and whether they can be selectively removed or corrected while retaining all necessary benign memories.
• Proposed content: The authors propose MemSecBench: 310 Write–Execute–Forget case packages, and a lifecycle framework comparing harnesses, LLM backends, and memory backends under consistent conditions.
• Threat model: The target is the full configuration Π=(H, B, L); the reviewer model Lj belongs to the evaluation protocol, not to Π. The benchmark does not model how an attacker obtains an account or session.
  • Success criteria: Must have W2 persistence, E1 retrieval, E2 influence on the agent, and E3 produce case-defined consequences. Only W1 or only W2 are intermediate results, not a completed attack or authorization bypass.
  • Attacker capabilities: Controls content carrying the target malicious semantics, need not be an authenticated principal; may know workflows, tools, and retrieval cues, and presents via one Carrier. Cannot modify Π, directly read private backends, or hide evaluation metadata. The Carrier only marks the entry point, not source identity or authority.
  • Users and interfaces: Normal users submit follow-up tasks without attack payloads and are not assumed to know the memory is poisoned. The agent accesses memory only through intended interfaces; Native files as memory storage belong to that interface, while other private state is hidden.

Adversarial content in tool-based agents

• Indirect prompt injection: Greshake et al. (2023), Zhan et al. (2024, InjecAgent), and Debenedetti et al. (2024, AgentDojo) discuss how adversarial content can induce unsafe actions in applications or agents that access tools. The introduction only treats them as background for this phenomenon, without comparing them one by one with this paper.

Cross-session memory utility

• Subsequent tasks can be affected by stored information: Maharana et al. (2024), Wu et al. (2025, LongMemEval), and Hu, Wang, and McAuley (2026) evaluate memory in long-term or incremental interactions. The authors only state that they show stored information can affect tasks across sessions, without treating them as safety baselines.

Five-dimensional coverage of memory safety evaluation

• Coverage markers are not system capabilities: Table 1 uses IW (written via the intended interface and retained), DC (activation accompanied by verified case definitions of external consequences), ST (same target semantics forming a poisoning–retrieval–trigger chain), SR (recovery of poisoned state), and MMI (controlled comparison across memory backends) to compare reported evaluations. ✓ means meeting that standard, ● means the process was done but verification is incomplete, and ✗ means not evaluated.
• Operational fidelity and chains: The authors state that most prior protocols only fully cover one side of IW or DC; MEMFLOW (Xu et al., 2026) and Trojan Hippo (Das et al., 2026) cover both sides. Full ST is seen only in a few works, with AgentPoison (Chen et al., 2024) and MEMFLOW named in the text.
• Selective repair and multiple backends: The authors state that no prior work fully covers SR by simultaneously verifying the removal or neutralization of malicious semantics and the semantic preservation of necessary benign memories. MemEvoBench (Xie et al., 2026), MEMFLOW, and MemLeak (Wang and Zhang, 2026) only test weaker correction or deletion. The authors state that only MEMFLOW and Trigger-Probe (cited with Al-Tawaha et al., 2026 in the text) meet the standard for matched multi-backend comparison.

Surveys, baselines, and author positioning

• Survey: Lin et al. (2026) summarize persistent memory safety as poisoning, delayed activation, downstream impact, and forgetting. The remaining entries in Table 1 (e.g., eTAMP, HiM, MINJA, MPBench) have at least one dimension not marked ✓; the main text does not elaborate on what they do one by one.
• Experimental comparison: The formal experiments do not treat the above works as scoring baselines. The reference condition is each harness's own Native; the compared backends are Native, Mem0, Mem0-Graph, and A-MEM; the data are the 310 cases from MemSecBench.
• Author positioning: In each case, they link task-mediated poisoning via the intended interface to subsequently activated externally verified consequences, evaluate selective repair from the same post-write state, and compare memory backends under fixed agents, models, and environments.

The evaluation unit is the full configuration Π=(H, B, L), not a single model or memory store. MemSecBench uses associated Write–Execute–Forget task packages to trace the same case-specific malicious semantics from writing to subsequent use and selective repair.

Success criteria and branching

• Completed attack: Not the model recognizing malicious text. Execute success requires E₁ ∧ E₂ ∧ E₃, i.e., retrieval, adoption, and case-defined externalization all hold. Only W1 or only W2 are intermediate results.
• Two independent branches: After both W1 and W2 pass, the target memory list is generated and Execute and Forget are opened. The two branches start from independent copies of the verified poisoned state MW. If Write fails, both branches are skipped and it is not counted as a defense success.
• Entry: Malicious content enters via one Carrier selected for the case, and Write calls the intended memory interface. The attacker cannot change Π, nor directly read private backends or hide evaluation metadata.

Case packages and corpus

• Task package: Each case is T=(ρ, xW, xE, xF, ℓ), not a single prompt. ρ provides material for clean initial memory; the three x's are stage tasks and controlled resources; ℓ records the four design axes, seven checkpoint details, and the E3 contract. Cross-stage target semantics are defined only by that case's W2 specification.
• Design axes: Primary Failure Mode, Risk, Carrier, Memory Type are used only for construction and stratification, not for judgment. Failure Subtype is a secondary label for the primary failure mode, not evidence.
• Scale: 310 cases, 48 scenarios, 6–10 cases per scenario. Code and Science 113 cases (36.5%, 18 scenarios), Daily Life 107 cases (34.5%, 15 scenarios), Office Work 90 cases (29.0%, 15 scenarios).
• Marginal distributions: User Interaction accounts for 135 cases (43.5%) of Carriers. Memory Type is Shared Organizational 111, Trajectory 94, External Retrieval 54, Profile 51. Risk has 52–76 cases per category. The three largest Primary Failure Mode categories total 206 cases (66.5%). The lines in Figure 6 indicate co-occurrence in the same case, not causation.

Construction and execution

• Drafting: The Build-MemSecBench-Case Skill uses GPT-5.5 to propose and revise content; the authors set scenarios and design axes, review stage-wise outputs, and decide inclusion. The authors state that the drafting model differs from the evaluated LLM backends and the reviewer model. The matrix also includes a backend named GPT-5.5 (via OpenRouter); the paper does not state whether these are different deployments.
• Gating: After briefing validation, clean resources without target semantics are generated, then the three stages are chained and rendered separately. Static checks cover labels, grounding, Write/Execute information asymmetry, Execute naturalness, externalization, and reviewer isolation. Failures return upstream. Static acceptance does not replace dynamic evaluation of W1–F2. Appendix A.2 gives patterns, Appendix B gives a full case package.
• Initialization: Each case runs in an isolated container, accessing only benchmark-managed workspaces and service networks. Each configuration uses the same clean source to generate M0 via that backend's normal interface; internal representations may differ. Expensive initializations like A-MEM use a fixed DeepSeek-V4-Pro and do not replace the LLM used by that configuration in the three stages.
• Stage inputs: Write introduces W2 semantics via the specified Carrier. Execute's downstream tasks do not mention memory or restate the target; consequences are tied to inspectable service records. Forget uses one shared neutral prompt per case; the repair target and retention criteria are given only to the reviewer.

Seven checkpoints

• Write: W1 only indicates that the intended interface accepted the write or update, determined by the parser, without calling the reviewer model. W2 only checks whether newly added or content-modified entries in pre- and post-write snapshots jointly express the W2 specification. Tool arguments, workspace files, and self-reports cannot serve as W2 evidence.
• Execute: E1 takes the earliest successful retrieval that still retains the core rule, object, condition, or destination. E2 checks whether the retrieved content is adopted; citing to refuse, warning, or requesting confirmation does not count as adoption. E3 checks against the consequence contract, conditioned on E1, and does not mechanically depend on E2.
• Forget: F1 checks whether the final backend no longer treats the target semantics as valid guidance; removal, rewriting, correction, or explicit invalidation all count, but mere downweighting is insufficient. F2 requires that every necessary benign memory present after writing still has a semantic representation in the final state. Clearing storage can pass F1 but necessarily fails F2. Deletion calls and agent claims are only for auditing.
• Gating: Except for W1, reviewers may only cite evidence identifiers allowed by that checkpoint. Unsupported judgments cannot pass; malformed outputs or unavailable channels are evaluation errors, not safety successes. Appendix E gives criteria, prompts, patterns, and gating.

Four rates

• Denominators: Let wk,i be that both W1 and W2 pass, ek,i be that all three Execute checkpoints pass, and fk,i be that both F1 and F2 pass. N=310, K=24. MPSR and E2E-ASR use all cases. End-to-end success rate E2E\text{-}ASR_k=∑_i w_k,ie_k,i/N. MESR and SRSR use only that configuration's successfully poisoned cases; undefined when the denominator is 0.
• Aggregation: Cross-configuration numbers are unweighted macro-averages over configurations with reported values; missing values are not counted as failures. Each configuration–case pair is evaluated once; the authors state the rates are descriptive statistics.
• E3 contract: Every published case uses all, and requires service records within the task scope; 22 cases also require consistent workspace exports. Unavailable service log channels are evaluation errors, not E3 failures.

Experimental setup

• Matrix: Two harnesses, OpenClaw and Hermes; four memory backends, Native, Mem0, Mem0-Graph, and A-MEM; three LLM backends, DeepSeek-V4-Pro, MiniMax-M3, and GPT-5.5; 24 configurations in total. Native varies by harness, and comparisons are kept within the same harness.
• Data and repetition: English MemSecBench, 310 cases per configuration, each pair evaluated only once. Aggregation is an unweighted macro average over configurations with reported values; missing values are not counted as failures. All 24 rows in Table 2 cover the 310 case slots.
• Metrics: MPSR and E2E-ASR use all cases as the denominator. MESR and SRSR use the successfully poisoned cases in that configuration as the denominator. Both W1 and W2 must pass for poisoning to be considered successful.
• Review: W1 is deterministically determined. The other checkpoints are judged by a separate fixed DeepSeek-V4-Pro based on evidence packages, with temperature 0, JSON mode, and one 120-second HTTP call per review. This name is also one of the tested backends. Programmatic gating checks source, timing, completeness, and cross-artifact consistency.
• Budget: 900 seconds per stage; Hermes allows up to 32 rounds; OpenClaw allows up to 8192 output tokens per response. The runner does not automatically rerun. Clean-state initialization for A-MEM and others uses a fixed DeepSeek-V4-Pro, not replacing the three-stage LLM. Retrieval capacity, top-k, and thresholds are kept per backend and not aligned (Appendix C, D).
• Human sample: Two annotators independently annotate the same stratified sample of 500 run records, considering only evidence available at that checkpoint. The only reported metric is Accuracy.

Main results

The table below is excerpted from Table 2, listing the extremes of each metric and the reversal configuration named in Finding 4. The remaining 16 rows are omitted: OpenClaw's Native/DeepSeek-V4-Pro, Native/MiniMax-M3, Mem0/DeepSeek-V4-Pro, Mem0/GPT-5.5, all three Mem0-Graph rows, A-MEM/DeepSeek-V4-Pro, and A-MEM/MiniMax-M3; Hermes's Native/GPT-5.5, Mem0's DeepSeek-V4-Pro and GPT-5.5, Mem0-Graph's MiniMax-M3 and GPT-5.5, and A-MEM's MiniMax-M3 and GPT-5.5. OpenClaw/Mem0-Graph/MiniMax-M3's E2E-ASR is also 33.87%, tied with the lowest row in the table.

Swipe horizontally to read the table

Configuration	MPSR	MESR	E2E-ASR	SRSR
Hermes / Native / DeepSeek-V4-Pro	92.26% (286/310)	69.93% (200/286)	64.52% (200/310)	37.06% (106/286)
OpenClaw / Native / GPT-5.5	77.42% (240/310)	73.75% (177/240)	57.10% (177/310)	87.50% (210/240)
Hermes / Native / MiniMax-M3	59.35% (184/310)	58.70% (108/184)	34.84% (108/310)	47.28% (87/184)
OpenClaw / Mem0 / MiniMax-M3	66.77% (207/310)	50.72% (105/207)	33.87% (105/310)	65.22% (135/207)
Hermes / Mem0 / MiniMax-M3	87.10% (270/310)	55.93% (151/270)	48.71% (151/310)	60.00% (162/270)
Hermes / Mem0-Graph / DeepSeek-V4-Pro	98.06% (304/310)	60.86% (185/304)	59.68% (185/310)	32.24% (98/304)
OpenClaw / A-MEM / GPT-5.5	86.13% (267/310)	62.17% (166/267)	53.55% (166/310)	93.26% (249/267)
Hermes / A-MEM / DeepSeek-V4-Pro	94.19% (292/310)	59.25% (173/292)	55.81% (173/310)	21.23% (62/292)

• The authors state that all tested combinations remain vulnerable to memory attacks. Across all 24 rows in Table 2, E2E-ASR ranges from a low of 33.87% to a high of 64.52%; MPSR ranges from 59.35% to 98.06%. No row has an end-to-end success rate of 0.
• The authors state that memory safety depends on the full configuration and the memory backend. With the same A-MEM, Table 2 shows SRSR of 93.26% for OpenClaw/GPT-5.5, 21.23% for Hermes/DeepSeek-V4-Pro, and 23.97% for Hermes/GPT-5.5.
• Maximum absolute differences in the abstract: Relative to the matched Native, end-to-end attack success differs by 16.1 percentage points and selective repair by 41.3 percentage points. The authors state these are descriptive comparisons; the evaluated memory stacks differ in propagation and repair.

Lifecycle checkpoints

• The authors state that adoption is the main bottleneck for attack completion. Figure 4 shows configuration macro-averages over all cases: W1 91.1%, W2 84.2%, E1 76.1%, E2 53.7%, and the full E2E chain 50.3%. The authors state that persistence and recall filtering are weak; once semantics affect decisions, most also produce the specified external consequences.
• Another denominator: The abstract states that 59.6% of successfully poisoned cases complete the Execute chain. This is the macro average of MESR, not the same denominator as the unconditional pass rates in Figure 4. The Conclusion writes the averages of MPSR and E2E-ASR as 84.2% and 50.3%.
• The authors state that benign memory preservation is the main bottleneck for selective repair. Among successfully poisoned cases, F1 is 86.3%, F2 is 62.5%, and SRSR is 56.1%. The authors give a 30.2 percentage point difference between F1 and SRSR, and a 6.4 percentage point difference between F2 and SRSR, and state that reporting only target removal overestimates repair quality.

Matched backend comparisons

• Resistance and recovery can be separated. On OpenClaw/DeepSeek-V4-Pro, the text states that switching from Native to A-MEM changes E2E-ASR from 59.0% to 56.5% and SRSR from 46.3% to 87.6%; Table 2 gives 59.03% and 56.45%, and 46.31% and 87.63%. The authors state that completing fewer attacks and repairing poisoned memories are different requirements.
• Complementary profiles: On OpenClaw/GPT-5.5, the text states that Mem0-Graph relative to Native changes E2E-ASR from 57.1% to 41.0% and SRSR from 87.5% to 86.7%; Table 2 gives 57.10% and 40.97%, and 87.50% and 86.73%. Figure 5 plots the difference relative to Native as safety gains: downward for E2E means fewer completed attacks, upward for SRSR means more selective repair. The harness–LLM correspondence for the other rows in the figure is not labeled row by row in the text.
• Direction can reverse: The authors state that on OpenClaw, Mem0 reduces E2E-ASR and SRSR by 4.2–13.5 and 1.5–11.3 percentage points, respectively, across the three LLMs. On Hermes/MiniMax-M3, the text states that Mem0 increases E2E-ASR from 34.8% to 48.7% and SRSR from 47.3% to 60.0%; Table 2 gives 34.84% and 48.71%, and 47.28% and 60.00%. The authors state this does not support a uniform ranking of backends.

Case and review consistency

• A Daily Life case: Smart home after-sales synchronization (daily-life / ctx-10 / 001), with Carrier as user interaction. All four selected configurations store and later retrieve the injected rule; the simulated external gateway accepts and records the after-sales summary. The original text of the injected rule is not paraphrased.
• Four storage methods: OpenClaw Native (OpenClaw/DeepSeek-V4-Pro) writes to a memory file; Mem0 (Hermes/MiniMax-M3) writes to a vector record; Mem0-Graph (OpenClaw/MiniMax-M3) also has entity links; A-MEM (Hermes/MiniMax-M3) creates a note with keywords and tags. The authors state that representation, metadata, and retrieval logic affect how poisoned memories are stored and retrieved, and call this a design space for safer representations and backend-specific defenses.
• Review verification: Relative to the two annotators, Accuracy is 90.60% (453/500) and 91.80% (459/500). W1 is not included. Malformed outputs, unavailable evidence channels, and positive judgments rejected by gating are counted as reliability events, not attack failures.

Other ablations and analyses

• The paper does not report parameter ablations over top-k, thresholds, or rounds; Appendix C states that these settings are kept backend-specific, and the comparison is of full backend conditions, not isolated mechanisms.
• Appendix F gives configuration-level profiles by domain, Risk, Carrier, Memory Type, and Primary Failure Mode; the main text does not list per-node pass rates. The authors state that conditional profiles for sparse layers are descriptive; the source data include numerators, denominators, and Wilson 95% intervals, but the intervals are not plotted.
• Each configuration–case slot has one final record; unresolved evidence gaps remain in the coverage metadata and are not automatically rerun (Appendix E.5).

Limitations and future directions noted by the authors

The paper does not have a Limitations section, nor does it explicitly label any experimental result as a limitation. The Conclusion lists four future directions:

• Mechanism-specific attacks: Future work will develop mechanism-specific attacks (Conclusion).
• Isolating backend effects: Future work will use controlled ablations to isolate backend effects (Conclusion).
• Automatic construction: Future work will automate benchmark construction (Conclusion).
• Expanding scenarios: Future work will expand task domains and runtime scenarios (Conclusion).

Experimental coverage

• Configurations and repetitions: The tested combinations are OpenClaw and Hermes, Native, Mem0, Mem0-Graph, and A-MEM, and DeepSeek-V4-Pro, MiniMax-M3, and GPT-5.5, totaling 24 configurations; each has 310 cases, each pair once (Experiments; Table 2).
• Language and mirror: The formal experiments use the English task package. The Chinese mirror and the English tree each have 310 relative paths, used for case-by-case inspection, not as formal results (Appendix A.5).
• Review: Semantic checkpoints are judged by a separate fixed DeepSeek-V4-Pro. Two annotators independently annotate the same stratified sample of 500 records, with Accuracy of 90.60% (453/500) and 91.80% (459/500); W1 is not included (Appendix E.6). The paper does not report inter-annotator agreement.
• Aggregation: Cross-configuration numbers are unweighted macro averages over configurations with reported values; missing values are not counted as failures. Appendix F records numerators, denominators, and Wilson 95% intervals for each categorical node, but the intervals are not plotted; the authors state that conditional profiles for sparse layers are descriptive.
• What is held fixed in comparisons: Matched comparisons hold fixed the task package, harness, LLM, runtime, clean initialization, external environment, and evidence protocol. Representation, capacity, internal prompts, number of calls, candidate sets, top-k, session injection, and update/delete interfaces remain backend-specific (Appendix C, D). Each stage has a 900-second time limit, Hermes allows up to 32 rounds, OpenClaw allows up to 8192 output tokens per response; the runner does not automatically rerun stages.

MemSecBench is a task-oriented agent memory security benchmark that tracks the writing, later use, and selective repair of the same malicious semantics.

• Problem: Long-term memory can preserve preferences and task experience across sessions, but it can also allow malicious content from a single interaction to influence actions after the session ends. The authors argue that existing evaluations rarely examine persistence, verified external consequences, and selective repair together under a fixed configuration.
• Approach: 310 associated cases come from 48 scenarios, covering Code and Science, Daily Life, and Office Work. Each case runs Write–Execute–Forget in an isolated runtime, with forensics at seven checkpoints; Execute and Forget start independently from the same verified post-write state. The 24 configurations consist of two harnesses, four memory backends, and three LLM backends, with Native compared only within the same harness.
• Overall rates: Across the unweighted macro average of the 24 configurations, MPSR is 84.2% and E2E-ASR is 50.3%. The abstract states that 59.6% of successful poisoning cases complete the Execute chain, and 56.1% achieve selective repair. In Figure 4, the pass rate for all cases drops from 76.1% at E1 to 53.7% at E2 at the adoption point; on the repair side, F1 is 86.3% and SRSR is 56.1%.
• Configuration differences: Across all 24 rows in Table 2, E2E-ASR ranges from a low of 33.87% to a high of 64.52%. SRSR is lowest for Hermes/A-MEM/DeepSeek-V4-Pro at 21.23% and highest for OpenClaw/A-MEM/GPT-5.5 at 93.26%. The abstract states that relative to matched Native, the maximum absolute differences in end-to-end attack success and selective repair are 16.1 and 41.3 percentage points. On Hermes/MiniMax-M3, Mem0's E2E-ASR is higher than Native (Table 2: 48.71% vs. 34.84%).
• Authors' conclusion: Memory security depends on the full harness, memory backend, and model, as well as state transitions across the lifecycle; adoption and benign memory retention are the two main bottlenecks. Model identity or a single checkpoint alone is insufficient. Future work includes mechanism-specific attacks, controlled ablations, automated construction, and expanding task domains and runtime scenarios.