Skip to content
Jasmine Wang· @j_asminewang · X·Indexed by this site · Originally published· Governance →· Its place in the scan record →

New California AI safety report recommends emergency shutdown authority and a cap on compute for recursive self-improvement

AI summary

An expert panel convened by California Governor Newsom releases an AI safety report, recommending granting emergency shutdown authority, setting minimum safety standards for AI developers, mandating third-party oversight, and authorizing a cap on compute used for recursive self-improvement. The report also recommends verified compute disclosure for automated AI R&D, mandatory loss-of-control reporting, and expanded whistleblower protections. The author, Jasmine Wang, calls it the most serious policy proposal on recursive self-improvement she has seen from any government, and says AI developers appear to be heading toward achieving recursive self-improvement within 6-12 months, and hopes the state legislature will move it forward next month.

Full text · Original

California’s new AI safety report is the most serious policy proposal for recursive self-improvement from any government I’ve seen. Very exciting!

It recommends the procedural machinery to actually slow down if & when we need to: verified compute disclosures for automated AI R&D, standby authority to cap that compute, emergency shutdown powers, mandatory loss-of-control reporting, and broadened whistleblower protections.

You can’t pace what you can’t see. If implemented, this would let the state see, and then act.

I hope the legislature moves on this when it’s back next month—AI developers seem on track to recursive self-improvement within 6-12 months, so we need these mechanisms practiced and working ASAP.

Quoted postNathan Calvin@_NathanCalvin
The recommendations from Governor Newsom's recently convened expert panel - Professor Alondra Nelson (Princeton), Jason Goldman, Professor Robert Reich (Stanford), and Professor Gillian Hadfield (Johns Hopkins) - were just released and are a clarion call for urgent action to address risks to public safety from advanced AI systems. Some of the most notable recommendations: • Emergency shutdown authority • Minimum safety standards for AI developers • Mandatory third party oversight • Authority to cap the amount of compute used for RSI The next weeks and months will determine whether these recommendations are words on paper, or whether the Governor and legislature can work together to accomplish something historic to protect CA and lead the country on guardrails to prevent catastrophic risks from runaway AI development.
View the quoted post on X

Source: Jasmine Wang · x.com