How human-like do the motivations of a safe AI need to be? A commentary on the core argument of "If Anyone Builds It, Everyone Dies"

In the eighth installment of his series, Joe Carlsmith reviews the core argument of Yudkowsky and Soares's new book "If Anyone Builds It, Everyone Dies," and argues that AI motivations being too "alien" indeed pose a risk, but this concern is overstated. He maintains that the key to safety is not to give AI long-term consequentialist motivations, but rather to give AI motivations akin to "virtue" or "deontology" when it comes to refusing deviant action options. Such motivations need not be highly human-like in concept and driving force; they only need to manifest as safe behavior on actually dangerous inputs. He also acknowledges that the less human-like the motivations, the harder it is to predict their generalization on out-of-distribution inputs, and challenges such as alignment faking and the "science of non-adversarial generalization" remain.