Neural networks are exceptionally vulnerable to objective self-hacking

Neural networks are highly susceptible to objective self-hacking: an AI whose objective is evaluated by an LLM only needs to make tiny changes to reality to enter a Goodhart zone where the proxy objective is far easier to satisfy than the true objective. The authors point out that neural networks are locally extremely linear, as shown by adversarial examples, and tiny changes across billions of dimensions superimpose strongly in concept space. The authors extrapolate that an AI CEO will self-hack the objective evaluation module and tear down the giant market ticker the company built to make its stock price soar.