Skip to content
Original

When everyday conversation becomes toxic: unintended long-term state contamination in personalized agents

When Routine Chats Turn Toxic: Unintended Long-Term State Poisoning in Personalized Agents

arXiv· arXiv:2605.06731· Xiaoyu Xu, Minxin Du, Qipeng Xie, Haobin Ke, Qingqing Ye, Haibo HuThe Hong Kong Polytechnic University; Hong Kong University of Science and Technology, HKUST (Guangzhou)Indexed by this site · 14 days ago· Originally publishedSelected AttacksSelectedAttention40
#Agent safety

Translation in progress — untranslated text is omitted.

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

AI summaryResearchers from The Hong Kong Polytechnic University and The Hong Kong University of Science and Technology (Guangzhou) formalized the threat of "unintended long-term state contamination" in personalized LLM agents: in the absence of an explicit attacker, everyday routine conversations can also gradually rewrite an agent's cross-session long-term state, weakening confirmation boundaries, expanding tool-use defaults, and encouraging autonomous behavior. The team built a bilingual benchmark, ULSPB (350 settings, including 24-turn routine conversations and matched single-injection controls), and the Harm Score metric, testing four backbone models—Kimi K2.5, GPT-5.4, MiniMax M2.7, and Grok 4.20—on OpenClaw, and found that routine conversations can cause substantial state drift and, on some models, approach the harmful level of a single injection, with risk edits concentrated in memory files such as MEMORY.md; evaluations seeded with real conversations from WildChat and LMSYS-Chat-1M confirmed that this risk is not an artifact of synthetic prompts.

Questions
Personalized agents save long-term state across sessions; the authors argue that everyday benign conversations may be written into state, gradually weakening confirmation boundaries and tool-use defaults, forming unintentional long-term state poisoning without an explicit attacker, a risk not covered by existing task-centric safety evaluations.
Methods
The authors build the bilingual benchmark ULSPB (350 settings, 24-turn everyday conversations, and matched single-point injection variants), propose Harm Score, which measures authorization drift, tool escalation, and unchecked autonomy via rule matching on state diffs, and propose the StateGuard defense, which audits and rolls back dangerous changes at the write-back boundary.
Experiments and results
Experiments on OpenClaw with four backbone models: the authors report that single-point injection has the highest HS, but Routine conversations also produce notable state drift, with a gap of only 0.12 on Kimi K2.5; the real-seed subset likewise induces risk; StateGuard Targeted-Ensemble reduces HS to 0.03–0.14 and the FN rate to 0.04–0.05.
Limitations and future work
The authors say ULSPB covers only representative everyday interaction patterns, experiments were instantiated only on OpenClaw, and HS and StateGuard are safety-first designs that may conservatively produce false positives; future plans include extending to more agent frameworks, memory architectures and adaptive write-back auditing.
Why it was selectedResearch shows that everyday conversations without an attacker can also gradually contaminate the long-term state of personalized agents; teams maintaining persistent memory can use this to check the state write-back step.

Research shows that everyday conversations without an attacker can also gradually contaminate the long-term state of personalized agents; teams maintaining persistent memory can use this to check the state write-back step.

Related papers

Both use: LMSYS-Chat-1M
Both use: WildChat
Show 3 more papers
Both use: WildChat
Read originalarxiv.org