Interpretable Meta-Policies for Neural Decision-Making

The researchers propose NEXUS, a hierarchical reinforcement learning framework that combines neural skills with a symbolic meta-policy, to alleviate the difficulty deep reinforcement learning has in being interpretable and aligned due to the black-box nature of neural networks. The framework moves transparent reasoning up to high-level skills rather than atomic actions, avoiding the overly complex and unreadable policies that result from directly symbolizing low-level actions, and substantially reduces symbolic complexity without sacrificing environment performance. Relying on object-centric observations, the reward function and meta-policy can be designed through interaction with an LLM. The paper reports that NEXUS agents learn disentangled skills, are interpretable at high-level goals, match the goals described in game manuals more closely than deep reinforcement learning, exhibit less reward hacking, and are more robust to game simplifications. The authors also separately evaluate the effects of three meta-policy architectures—neural, symbolic and neurosymbolic—on environment return and performance on alignment tasks.