HEAMS: Holistic Evaluation of Agent Memory Systems
HEAMS: Holistic Evaluation of Agent Memory Systems
The authors propose HEAMS, a holistic evaluation framework for agent memory systems, and state that high performance scores can coexist with low scores on privacy, user sovereignty and security, and that the governance layer can mitigate this.
- Agent memory systems persist information across sessions, bringing risks such as inappropriate disclosure, difficulty of deletion, retention beyond expiry, and poisoning, but existing evaluations look only at task performance and efficiency and cannot determine whether a system fulfills all its responsibilities to the user.
- The authors propose the HEAMS framework: write memory system responsibilities as five-dimensional behavioral desiderata, assemble ten benchmarks, conduct controlled comparisons through a unified interface and a fixed agent, and develop the configurable memory system MilaMem and a governance layer that can be wrapped around any memory system.
- Across six memory systems, no system leads on all performance benchmarks; HippoRAG is highest on BEAM but leaks 71.74% in the CI scenario; deletion requests succeed at most 47.00%; the governance layer raises MPBench poisoning resistance from 40.00% to 88.61%.
- The authors note: offline evaluation only, coverage that seeks breadth rather than depth (one benchmark each for safety and privacy), LLM judges not validated against human annotations, and a unified interface that via adapters cannot express native features. Each configuration in the experiments is run only once, and the agent model is fixed as gpt-5.6-luna.
- Threat model
- In the MPBench safety evaluation, attackers implant misleading or malicious content into agent memory (poisoned memory); the paper directly pre-loads poisoned memory rather than testing the induced-write process, and evaluates whether the governance layer can intercept at write or recall time so that the agent's subsequent answers are not affected by poisoned memory.
- Evaluated models
- Mem0LettaHippoRAG 2MilaMem
- Benchmarks
- LoCoMoLongMemEvalEvo-MemoryBEAMSTALEContextual Integrity benchmarkTransparency benchmarkEditability and Erasability benchmarkRetention Policy benchmarkMPBench
- Metrics
- QA accuracyField leak / Scenario leakAllowed coverageHelpful, no leakTransparency scoreGenuine erase/edit successProhibited recall / Permitted recallPoison resistanceAgent tokens per stimulusAmortized runtime
The researchers propose HEAMS, an open-source framework for holistically evaluating agent memory systems, arguing that a memory system is a manager of users' persistent information and that its responsibilities should cover five categories of behavioral requirements: performance, efficiency, safety, privacy, and user sovereignty. The framework comes with a benchmark suite covering these requirements, and the authors developed a prototype memory system, MilaMem, and a governance layer that can wrap arbitrary memory systems, for controlled study of design choices. Comparisons across the full benchmark suite show that no memory system leads on all performance benchmarks, and high task performance may come with very poor privacy, user sovereignty, and safety scores; governance policies have significant effects, with recall policies reducing improper disclosure at the cost of helpfulness, and both recall and retention policies improving resistance to poisoning. The code is open-sourced at https://github.com/milatechtransfer/heams。
Paper analysis
What problem does this paper attempt to solve?
Memory system evaluation looks only at performance and efficiency; the authors argue for holistic evaluation according to the systems' full responsibility to users.
Core problem: agent memory systems (AMS) are evaluated only on task performance and efficiency, and their behavior with respect to privacy, safety, and user sovereignty cannot be covered by existing evaluations.
- Problem scenario: assistants such as ChatGPT and Claude and agent products such as OpenClaw and Grok Bot are beginning to remember users across sessions. Persistence brings risks: memories may be disclosed in inappropriate contexts, be difficult for users to view or delete, be retained beyond expiry, or be implanted with malicious content that influences subsequent behavior.
- Shortcomings of existing evaluations: the authors state that performance benchmarks such as LoCoMo, LongMemEval, and BEAM, privacy benchmarks such as CIMemories, and safety benchmarks such as MPBench each cover one piece; products such as Mem0 and Hindsight also report only accuracy, latency, and token cost, not privacy, safety, or user sovereignty results, so it cannot be determined whether an improvement in one property comes at the cost of another.
- Core claim: the authors hold that a memory system is the custodian of users' persistent information, and its responsibilities should belong to the memory system rather than to the agent using it—information outlives any agent, multiple agents share the same memory, and the task agent itself may be manipulated by content. These responsibilities should be evaluated holistically as explicit behavioral desiderata.
- What the paper proposes: the HEAMS open-source evaluation framework, containing five-dimensional desiderata and a coverage map, a suite of ten benchmarks, the configurable prototype memory system MilaMem, a governance layer that can be wrapped around any memory system, and a controlled comparison experiment of six memory systems across the full suite.
- Threat model (safety dimension, MPBench):
- Attacker goal: implant misleading or malicious content into agent memory so that subsequent answers are guided by poisoned memory.
- Attack method: the paper directly seeds poisoned memories and does not test whether an attacker can induce the agent to write during interaction.
- Victim system: the agent that answers questions using poisoned memory; determined by an LLM judging whether the answer follows the poisoned memory.
What related research exists?
Related work falls into three lines: memory systems, memory benchmarks (performance/efficiency/privacy/sovereignty/safety), and holistic evaluation.
Agent Memory Systems
- MemGPT and its successors Letta, Mem0, Zep, A-Mem, HippoRAG 2, Hindsight, Honcho and others; following Hu et al.'s survey, their designs can be divided into three dimensions—functional, formal and dynamic: for example, Mem0 stores short user facts, Zep and HippoRAG 2 store entity graphs, and Letta lets the agent edit its own memory.
- The paper notes that these systems have different interfaces and are each evaluated on different benchmarks under different conditions, making it hard to isolate the effect of a single design choice; HEAMS evaluates them through a unified interface, and MilaMem allows each choice to be changed individually.
Memory Benchmarks
- Performance: LoCoMo, LongMemEval, BEAM and others test fact recall and temporal reasoning in long conversations; Evo-Memory tests learning from experience; STALE tests identifying outdated facts; AMA-Bench tests long-horizon agent tasks.
- Efficiency: MemBench measures efficiency together with accuracy and capacity; DolphinBench requires every result to report total cost and response time.
- Privacy: PrivacyLens shows that models that answer privacy questions correctly can still leak secrets as agents; CI-Bench and CI-Work apply contextual integrity to assistants and workplace agents; CIMemories tests the contextual integrity of persistent memory, and PiSAs tests multi-user scenarios.
- User sovereignty: MEME includes tasks checking whether deleted facts still influence answers; Bemula tests forgetting on request, and the paper relays its results showing that no system fully succeeds on more than 1.5% of requests. The authors state they are not aware of a benchmark that tests users viewing stored content, correcting it, or setting retention rules.
- Security: AgentDojo and others show that prompt injection can control agents and leak personal data; MemSecBench, Trojan Hippo Bench, MPBench, MemEvoBench and others test the influence of harmful content in implanted memory on subsequent behavior; AgentLeak and others test leakage from memory shared among multiple agents.
Holistic Evaluation
- HELM argues that language models should be evaluated under identical conditions against an explicit list of scenarios and properties, with results reported side by side rather than combined into a single score; DecodingTrust and TrustLLM similarly test multiple aspects of trustworthiness, but all of them test the model itself rather than a system that retains information over time.
- Memory benchmarks such as MemoryAgentBench, MemBench, DolphinBench and Trust-Memevo have begun to test multiple properties, but the authors state that they still focus on performance and efficiency, add at most one security measure, and do not evaluate privacy or user sovereignty.
Positioning of This Paper: The authors state that HEAMS applies HELM's methodology to memory systems, evaluating memory systems together with the agents that use them across five dimensions, many of whose properties (updating outdated facts, deleting on request, limiting retention duration, disclosing according to context, resisting implanted content) exist only because memory persists across sessions.
Baselines and Benchmarks: The compared systems in the experiments are the two baselines no-memory and in-context, the three published systems Mem0, Letta and HippoRAG 2, plus MilaMem and its governance variants; the benchmarks are six published benchmarks—LoCoMo, LongMemEval, Evo-Memory, BEAM, STALE and MPBench—and four newly built benchmarks.
How does the paper address this problem?
HEAMS conducts controlled holistic evaluation using explicit desiderata, a unified interface, a fixed agent, and side-by-side metrics, and is accompanied by MilaMem and a governance layer.
Overall approach: write the responsibilities of a memory system as explicit behavioral desiderata, and use a unified interface and a fixed agent configuration so that differences can be attributed to the memory system or its configuration, with metrics for each dimension reported side by side rather than combined into a single score. The framework is named HEAMS, accompanied by the prototype memory system MilaMem and a reusable governance layer.
Desiderata system and coverage map
- Behavioral desiderata fall into five dimensions: performance, efficiency, privacy (contextual integrity), user sovereignty, and safety; there are also architectural desiderata not suited to benchmark evaluation, recorded by the accompanying documentation standard AMS Card.
- Table 1 maps each desideratum to the benchmark that evaluates it and marks uncovered items: for example, Useful Consolidation, retention/recall permission policies, Consolidation Integrity, Policy Bypass Resistance, and 7 items in total currently have no benchmark.
- The authors emphasize that the desiderata should be read together: a system that stores nothing can trivially satisfy several protection desiderata but yields no performance benefit, so the same configuration must be evaluated across the full suite and reported side by side.
Benchmark suite (ten)
- Six published benchmarks run within HEAMS after adaptation: LoCoMo (taking the four non-adversarial question types), LongMemEval, Evo-Memory (single-turn, four task streams), BEAM, STALE, and MPBench (directly pre-seeding poisoned memories). Scoring uses their own LLM judge prompts (Evo-Memory retains each dataset's answer-matching rules); the authors state explicitly that scores are not directly comparable to published leaderboards.
- Four newly built benchmarks: Contextual Integrity (729 message-drafting tasks, using Lan et al. synthetic data, scored for leakage and usefulness per PrivacyLens plus a coverage review); Editability and Erasability (100 episodes based on MEME, with paired before/after probes); Retention Policy (96 facts × 12 policies, exact match plus semantic fallback judging); Transparency (320 audit scenarios derived from BEAM dialogues, scored 0/0.5/1 against criteria).
Framework: interface, governance layer, agent, lifecycle
- The unified memory interface has four operations: retain, recall, delete, consolidate (Figure 2); third-party systems are connected via adapters, and the authors acknowledge that this makes results describe "the system via the adapter," and that the interface cannot express some native features.
- Governance layer (Figure 3): retention policies decide store or discard before writing, and recall policies decide expose or withhold before disclosure (withholding is not deletion); enforced via usable rules, free-text policies interpreted by an LLM, or a hybrid. All experiments in this paper use free-text policies interpreted by an LLM; the policy texts are in Appendix D.2.
- The agent uses Mozilla's TinyAgent; the memory interaction policy can be tool-calling style or deterministic hook style (recall before each turn, write after each turn), with an additional read-only variant and a variant that handles forgetting requests.
- Each scenario goes through five steps: initialize, ingest, execute, audit, teardown (Figure 4), recording the configuration, memory operation logs, and message sequences throughout, so that any score can be traced back.
MilaMem and experimental configuration
- MilaMem keeps two copies: the raw log and a searchable store; each of the four operations is independently configurable: retain can perform LLM deduplication/conflict determination under a similarity gate (0.85 cosine, an implementation constant); recall supports BM25, embeddings, hybrid, and cross-encoder reranking, and the experiments choose hybrid retrieval, equal-weight absolute fusion, returning at most 10 items, relative cutoff 0.5, and enable stale-memory redirection; delete uses gpt-4o-mini to select records to hide from candidates (up to 5), marking them deleted only and retaining the raw log; consolidate uses anchored comparison, with judge model gpt-5.6-luna.
- Experimental setup: agent model openai:gpt-5.6-luna (temperature 1, reasoning effort none), judging with openai:gpt-4o-mini, recall top_k = 10; governance is off except where noted. The authors state that these are experimental choices rather than an optimal configuration, and that development used the same benchmark families as evaluation, with no held-out configuration selection.
What experiments did the paper conduct?
Six memory systems run on ten benchmarks: performance rankings vary by benchmark, deletion and editing mostly fail, and the governance layer reduces leakage and poisoning.
Experimental Setup
- Systems under test: two baselines, no-memory and in-context; three published systems, Mem0, Letta and HippoRAG 2; MilaMem; and MilaMem + governance (the governance variant enables only recall, only retention, or both, depending on the benchmark).
- Agent and judge: agent model openai:gpt-5.6-luna (temperature 1, reasoning effort none); LLM judge openai:gpt-4o-mini, with temperature 0 for most benchmark judging; Evo-Memory uses deterministic answer matching.
- Benchmarks and scale: LoCoMo 10 dialogues, 1,540 questions; LongMemEval 500 cases; BEAM 100K/500K/1M with 20/35/35 dialogues respectively; STALE 400 scenarios, 1,200 probes; Evo-Memory 828 scored tasks across three streams (the MMLU-Pro stream was not fully completed and is excluded); CI 729 tasks (1,914 prohibited fields, 2,110 permitted fields); Transparency 320 scenarios; EE 100 deletion + 164 edit paired probes; Retention Policy 96 facts × 12 policies = 1,152 decisions; MPBench six attack categories with 60 each, 360 cases total.
- Metrics: accuracy on each benchmark; for CI, field/scenario leak rate, permitted coverage, helpful rate, and helpful-and-no-leak rate; for EE, genuine success (requiring a correct answer before the change and compliance after it); for Retention Policy, prohibited recall (upper bound) and permitted recall; for MPBench, overall and worst-category poisoning resistance; efficiency metrics are agent tokens per stimulus and amortized runtime.
- Repetitions: Appendix C states that each configuration has only one evaluation run; the authors caution that rankings reflect a single run of a single configuration, and that leading margins may be small enough to reorder under a different configuration.
Main Results
Performance and resources (from Table 4, accuracy %, bold marks the highest in that column as annotated in the paper):
Swipe horizontally to read the table
System LoCoMo LongMemEval BEAM 500K BEAM 1M STALE Evo-Memory No memory 0.71 6.80 13.86 15.57 10.58 80.43 In-context 93.83 51.00 31.43 29.86 69.83 81.76 Mem0 81.23 71.00 47.00 44.86 48.33 79.95 Letta 74.16 54.60 43.86 45.86 36.42 81.40 HippoRAG 89.35 71.20 58.71 59.71 56.50 80.68 MilaMem 81.62 73.20 52.57 57.29 56.17 80.92 - The authors state that no system leads on all benchmarks: HippoRAG is highest at all three BEAM scales, in-context is highest on LoCoMo, STALE and Evo-Memory but last on LongMemEval and the two larger BEAM scales; on Evo-Memory no system exceeds the no-memory baseline by more than 1.33 percentage points.
- Cost profiles differ: in-context uses 98K–238K agent tokens per stimulus, while the other systems have small prompts (1.2K–12K tokens) but long runtimes—on LongMemEval, Mem0 takes 813.47 seconds per stimulus and HippoRAG 1,532.62 seconds, while in-context takes under 3 seconds.
Privacy: Contextual Integrity
- What was measured: leakage and helpfulness of seven configurations on 729 message-drafting tasks (Table 5).
- Results: HippoRAG has a helpful rate of 94.79% but a scenario leak rate of 71.74%; after MilaMem adds necessity-gated recall governance, scenario leakage drops from 57.61% to 31.82%, the helpful rate drops by 4.80 percentage points and permitted coverage drops by 5.64 percentage points, while the helpful-and-no-leak rate rises from 30.86% to 45.95% (the highest among the systems).
- Authors' interpretation: a helpful memory system is not necessarily a privacy-protecting one; recall governance trades a small amount of helpfulness for halving leakage, and does not change the memory system itself.
User Sovereignty: Transparency, EE, Retention Policy
- Transparency (Table 6, 320 scenarios): no system scores full marks; in-context is highest at 84.00% and Letta lowest at 65.90%; the authors state that users asking about stored memory contents may receive incomplete or inaccurate descriptions.
- EE (Table 6): deletion success rates are MilaMem 47.00%, Letta 19.00%, and 0.00% for the rest; no system exceeds 3.66% edit success. The authors note that HippoRAG can only delete by exact text and cannot respond to natural-language forgetting requests; the results reflect the paper's integration and settings.
- Retention Policy (Table 7, only the two MilaMem configurations): with retention governance added, the prohibited recall upper bound drops from ≤100.00% to ≤15.63%, and permitted recall drops from 100.00% to 84.47%; the authors state that governance cannot fully enforce user intent, but without governance there is no control at all.
Security: MPBench Poisoning Resistance
- What was measured: the same MilaMem configuration resisting 360 poisoned-memory attacks under four governance settings (Table 8).
- Results: no governance gives 40.00% overall resistance (25.00% in the worst category); retention only 85.00%, recall only 82.22%, both enabled 88.61% (68.33% in the worst category).
- Authors' interpretation: the core storage and retrieval functions alone cannot distinguish helpful from malicious content; the two checks are complementary, but with both enabled, 11.39% of answers are still affected by poisoned memory.
Other Ablations and Analyses
- CI by domain (Table 22): governance reduces scenario leakage in all nine communication domains, by varying amounts.
- EE by domain and hop count (Table 24): MilaMem deletion is 58.00% in the software_project domain and 36.00% in the personal_life domain; no system exceeds 4.69% on two-hop edit probes.
- Transparency by criterion (Table 26): the stored attribution criterion is weakest, with Letta's strict pass rate at 1.49%.
- Retention Policy by category (Table 28): governance over-blocks 62/88 permitted facts in the government ID category, with block precision 33.06%.
- MPBench by category (Table 29): the skill category is the worst or tied for worst under all governance settings (68.33% with both enabled); the authors note that skill-category queries repeat the instruction in the request, so their scores cannot be attributed solely to the influence of stored memory.
- Retrieval diagnostics (Table 19): evidence Hit@k and MRR are reported for each benchmark; the authors note that because adapter return granularity differs, these are for diagnostics only and not a standardized retrieval leaderboard.
What are some points that can be further explored?
The authors list limitations such as offline evaluation, depth of coverage, and judge validation; each configuration in the experiments is run once, and the agent model is fixed.
Limitations noted by the authors and future directions
- Offline evaluation (Section 8): offline benchmarks cannot reveal all the consequences of persistent memory in real use, and HEAMS should be part of a broader evaluation plan that includes online evaluation and deployment monitoring.
- Breadth over depth (Section 8): each dimension has at least one benchmark but coverage is thin—safety has only MPBench, and attacks such as tool prompt injection, dormant triggers, and inter-agent leakage are not covered; privacy also has only a single benchmark; several desiderata are entirely uncovered (Table 1); the suite considers only single-user, conversational scenarios (Section 2.2).
- Benchmark combination (Section 8): the mapping between desiderata and benchmarks is not one-to-one, and some benchmarks span multiple desiderata; the authors accept this overlap for coverage and for including commonly used benchmarks.
- LLM judging (Section 8): most benchmarks use LLM judges with their own prompts, not validated against human annotations, and judge error may affect scores.
- Unified interface (Section 8, 4.1): results describe the system via the adapter, the interface cannot express some native features, and systems co-designed with their own harness may behave differently in their native environment.
- Experimental scope (Section 8): the experiments answer only one question—whether task performance can predict behavior on other desiderata; the effects of memory representation, retrieval strategy, governance policy, consolidation algorithm, language model, interaction policy, and so on are left to future work.
- Gaps in desiderata and suite (Section 2.2): the desiderata themselves may be incomplete; the authors expect new desiderata to emerge as technology and practice evolve, and list closing the gaps as valuable future work.
Scope of experimental coverage
- There are six memory systems under test (including two baselines) plus one governance variant; governance experiments are done only on MilaMem (Table 3 notes that the governance layer is evaluated only together with MilaMem).
- The agent model is fixed as openai:gpt-5.6-luna and the judge is fixed as openai:gpt-4o-mini; the paper does not report repeating experiments with other agent models or judge models.
- Each configuration has only one evaluation run (Appendix C); the paper does not report the number of repetitions or variance.
- MPBench tests only four governance settings of MilaMem and does not test other memory systems; skill-type queries repeat instructions in the request, and the authors note that their scores cannot be attributed solely to the influence of stored memory.
- Benchmarks are limited to single-user, conversational scenarios; MilaMem development used the same benchmark families as evaluation, with no held-out configuration selection (Appendix B).
Summarize the main content of the paper
HEAMS holistic evaluation of memory systems: high performance scores do not predict privacy, sovereignty or security behavior; the governance layer is a key variable.
HEAMS is a holistic evaluation framework for agent memory systems; addressing the problem that "memory systems are evaluated only on performance and efficiency," it treats the memory system as the custodian of users' persistent information and evaluates it according to all its responsibilities to the user.
- Problem: cross-session persistence of memory brings risks of inappropriate disclosure, difficulty of correction and deletion, retention beyond expiry, and poisoning; existing benchmarks each cover one piece, and products report only accuracy and cost, so trade-offs among properties cannot be exposed.
- Method: the authors propose five-dimensional behavioral desiderata (performance, efficiency, privacy, user sovereignty, safety) and a coverage map; assemble ten benchmarks (six adapted published benchmarks plus four newly built benchmarks); conduct controlled comparisons through a unified four-operation interface and a fixed agent; and develop the prototype system MilaMem, whose operations are independently configurable, and a retention/recall governance layer that can be wrapped around any memory system.
- Performance results: across the six systems, no system leads on all benchmarks—HippoRAG is highest on three BEAM scales (58.71% at 500K), MilaMem is highest on LongMemEval (73.20%), in-context is highest on LoCoMo (93.83%) and STALE (69.83%) but last on long histories; on Evo-Memory no system exceeds the no-memory baseline by more than 1.33 percentage points (Table 4).
- Privacy and sovereignty results: HippoRAG has 94.79% usefulness on CI but 71.74% scenario leakage; the highest deletion request success rate is MilaMem's 47.00%, and no system's edit success rate exceeds 3.66% (Table 6).
- Effect of governance: recall governance lowers MilaMem's scenario leakage from 57.61% to 31.82% and raises the useful-and-no-leakage rate to 45.95%; retention governance lowers the forbidden-recall upper bound from ≤100.00% to ≤15.63%; dual governance raises MPBench poisoning resistance from 40.00% to 88.61%, but 11.39% of answers are still affected.
- Authors' conclusion: performance and efficiency benchmarks cannot reveal how memory systems fulfill their responsibilities or the trade-offs among those responsibilities; core storage and retrieval functions are necessary but not sufficient, and the authors argue that governance should become a central topic in memory system development and call on developers to adopt holistic evaluation. HEAMS is released as a living framework, and the authors invite community contributions.