Cooperative Inverse Reinforcement Learning (CIRL)
Cooperative Inverse Reinforcement Learning
Quick read
Based on the paper's full text (AI-generated); refer to the original
Hadfield-Menell, Dragan, Abbeel, and Russell of UC Berkeley propose cooperative inverse reinforcement learning (CIRL), formalizing value alignment as a cooperative partial-information game between a human and a robot, where both receive payoff according to the human's reward function, but the robot initially does not know that function. The paper proves that computing the optimal joint policy can be reduced to solving a POMDP, that the robot's posterior over reward parameters is a sufficient statistic, and that the complexity is exponentially lower than that of a general Dec-POMDP; it also argues that an expert demonstration that is optimal in isolation (the assumption of classical IRL) is not optimal in CIRL, and that the optimal solution produces active teaching, active learning, and communicative actions. In 2D grid navigation experiments, when the robot uses maximum-entropy IRL, the authors' approximate teaching demonstration strategy outperforms expert demonstrations on all metrics (F>962, p<.0001), and regret is minimized when the robot's expectation of the human's optimality takes an intermediate value. The authors argue that rather than writing a fixed objective into the machine, one should design machines that can provably converge to the correct objective through interaction.
- Autonomous systems need to align their own values with human values in order to be both useful and not bring risk; but humans often cannot articulate their goals clearly, and classical IRL assumes human behavior is isolatedly optimal, so it cannot characterize teaching behavior.
- Formalizes value alignment as cooperative inverse reinforcement learning (CIRL): the human knows the reward parameters θ and the robot does not, and both share the same reward; the authors reduce the computation of the optimal joint policy to a POMDP and give an approximate CIRL algorithm under linear rewards.
- Proves that the robot's posterior over θ is a sufficient statistic, and that isolated optimality (DBE/IRL) is generally suboptimal in CIRL; in grid navigation experiments, teaching-style demonstrations let the robot infer a reward closer to the true one, with regret significantly lower than with expert demonstrations, and regret is lowest when the robot's expectation of human optimality takes an intermediate value.
- The optimal joint policy assumes both parties can coordinate perfectly in advance, which is infeasible in reality; the coordination problem of multiple optimal policies is left for future work, and the authors propose that research should study how two independent agents can reach mutual best responses.
The paper formalizes value alignment as a cooperative game in which both the human and the robot receive payoff according to the human's reward, and argues that an expert demonstration that is optimal in isolation is not an optimal teaching strategy.