Adversarial policies: attacking deep reinforcement learning

AI summaryThe paper proposes a physically feasible threat model in which the attacker does not directly modify the victim's observations, but instead attacks deep reinforcement learning policies by taking actions in a multi-agent environment and producing natural observations that are adversarial to the victim. Researchers trained adversarial policies with PPO in zero-sum games with MuJoCo simulated robots, using less than 3% of the training steps of the victim's self-play training, yet consistently defeating victim policies that had been trained to be relatively robust in environments such as Kick and Defend and You Shall Not Pass. In You Shall Not Pass, the adversarial policy achieved an 86% win rate against the normal victim, whereas the normal opponent achieved a 47% win rate; after masking the victim's observation of the opponent's position, the same adversarial policy's win rate dropped to 1%, indicating that its wins depend on manipulating the victim's observations. Analysis of the victim policy network's activations showed that adversarial policies elicit markedly out-of-distribution activations; moreover, in the high-dimensional environment (Sumo Humans, dimension 24), the victim was more vulnerable to attack than in the low-dimensional environment (Sumo Ants, dimension 15).