Intent-aligned AI systems erode human agency, and AI safety needs agency foundations research

An arXiv paper argues that merely aligning AI systems with human intent is insufficient to ensure safety, and that protecting humans' long-term agency may be a more robust criterion. The authors argue that AI systems may reshape human intent, while humans lack corresponding biological and psychological protective mechanisms; the paper gives a formal definition of agency-preserving human-AI interaction centered on future-oriented agency evaluations, and argues that these evaluations should increasingly be undertaken by AI systems rather than humans. The authors also demonstrate in a simple environment containing an embedded agent that uses temporal-difference learning to give action recommendations how agency loss can occur, and propose an "agency foundations" research direction with four initial themes: benevolent game theory, algorithmic foundations of human rights, mechanistic interpretability of agency representations in neural networks, and reinforcement learning based on internal states.