Discovering agents: a causal definition of agents and a discovery algorithm

The paper proposes a formal causal definition of an agent: if an agent adjusts its policy when its actions affect the world in different ways, then it is an agent; the authors state that this is the first formal causal definition of an agent. Based on this definition, the authors give a causal discovery algorithm for discovering agents from empirical data. They also give algorithms for converting between causal models and game-theoretic influence diagrams, and use this method to clarify confusions previously caused by improper causal modeling of agents. The authors point out that causal models have often been directly assumed by modelers without justification, and that modeling errors can lead to errors in safety analysis.