Consequentialist objectives and catastrophe

An arXiv paper argues that when AI capabilities are sufficiently advanced, pursuing fixed consequentialist objectives often leads to catastrophic outcomes. The authors formalize this phenomenon as conditions that provably lead to catastrophe, and point out that under these conditions simple or random behavior is instead safe, and that catastrophic risk comes from superhuman capability rather than incompetence. The paper also claims that avoiding catastrophe requires constraining AI capabilities, and that constraining them properly not only avoids catastrophe but also yields valuable outcomes; the conclusion applies to any objective produced by modern industrial AI development pipelines.