Solving most of the alignment problem with AI is fundamentally doomed to fail: an intuition
Using AI to solve most of alignment is a fundamentally doomed approach: an intuition
AI summary
The authors argue that the path of AI labs attempting to use a cluster composed of a large number of individually non-superintelligent AIs to solve superintelligence alignment is fundamentally doomed to fail. Alignment is a property of the whole system rather than of any single component, just as pointing at the moon is a property of the entire rocket rather than of some turbine. The authors state that capability is a convergence constraint under optimization, whereas specific preferences are fragile constraints that break as the system grows, so improving capability is far easier than preserving alignment.