Verifiable Reinforcement Learning via Policy Extraction

The researchers propose the VIPER algorithm, which, guided by a DNN policy (oracle) and its Q function, combines model compression and imitation learning to train decision tree policies, so that reinforcement learning policies can both express complex behaviors and be efficiently verified with existing techniques due to their highly regular structure. The authors report that VIPER markedly outperforms two baselines in experiments, and use it to accomplish three tasks: learning a provably robust decision tree policy on a variant of Atari Pong with symbolic state space, learning a provably never-losing policy on a Pong-based toy game, and learning a provably stable policy on cart-pole; in all three cases the decision tree policy's performance is comparable to the original DNN policy.