PolicyBank: Enabling LLM Agents to Evolve Policy Understanding

Researchers propose PolicyBank, a memory mechanism that maintains structured tool-level policy insights and iteratively revises them through corrective feedback from interaction and pre-deployment testing, enabling LLM agents to autonomously correct policy interpretations, closing up to 82% of the gap to a human oracle in policy-gap scenarios. The paper points out that organizational policies specify authorization constraints in natural language and inevitably contain ambiguity and logical or semantic gaps, causing agent behavior to systematically deviate from the true requirements; existing memory mechanisms treat policy as an immutable baseline, which reinforces "compliant but wrong" behavior. The authors also extend a commonly used tool-calling benchmark, constructing a test set with controlled policy gaps to separate alignment failures from execution failures; existing memory mechanisms achieve a success rate close to zero in this scenario.