Memory tax: Dissecting the net value of agent memory

A large-scale evaluation covering multiple benchmarks and model backbones finds that many specialized memory systems for long-horizon agents perform worse than naive RAG in many cases, with a negative net value. The paper decomposes the net value of a memory system relative to directly retrieving the original history into memory benefit and memory tax; memory processing reorganizes facts or distills preferences to help the task, but may also delete or obscure evidence, thereby incurring a memory tax. The authors focus on the causes of the neglected memory tax on both the system and evaluation sides, and generalize the concept to more memory application scenarios. They plan to open-source the code and results, with the project page at https://hexuan21.github.io/MemoryTax/。