Responsive Safety in Reinforcement Learning via PID Lagrangian Methods

The authors propose a PID Lagrangian multiplier update method using the derivative of the constraint function to mitigate oscillation, overshoot, and constraint violations during safe reinforcement learning training. The method treats conventional multiplier updates as integral control and adds proportional and derivative control to damp and anticipate learning dynamics, setting best results on the Safety Gym benchmark for deep reinforcement learning; it also provides a controller tuning method invariant to the numerical scales of reward and cost, and experiments show improved performance and hyperparameter robustness, with implementation close to conventional Lagrangian methods.