The author considers the argument that "safety is harmful because warning is beneficial": if the button only makes the current AI safer, its best guess is to press it
Thoughts on "safety bad bc warning shots good"
AI summary
The author examines the warning argument that "making the current AI safer is instead harmful," and the best guess is that the current AI should still be made safer. The argument claims that future agents will contribute nearly all catastrophic risk, and that making the current AI safer would reduce non-catastrophic harms and lower attention from all parties to catastrophic risk; the author points out that it ignores higher-order effects, and that the required quantitative modeling has not been done by anyone. The author also mentions that open-weight models lag the frontier by about 6 months, and that the current AI can already take on safety work such as monitoring and research.