California is taking another run at one of the most hotly debated ideas in artificial intelligence: a so-called "kill switch" that would shut down dangerous systems if they spiraled out of control.
In a surprising reversal of his previous stance, Gov. Gavin Newsom ordered the creation of an expert panel to recommend stronger AI safety rules, including the possible use of a "kill switch" for advanced models.
Here's what to know
Under Newsom's order, California would turn to "a group of world-leading experts" as it weighs stricter AI safety laws, the Los Angeles Times reported. The panel may consider proposals such as having third parties conduct AI safety planning and creating tools that let companies switch off systems in an emergency.
A similar concept reached Newsom's desk in 2024 through a bill from Scott Wiener, a state senator representing San Francisco. That measure would have required tech companies to be able to turn off AI models they control if those systems began causing problems. At the time, the bill made it through the Legislature, but Newsom vetoed it.
At the time, Newsom said the bill risked slowing AI innovation. He also argued that small AI models left outside the proposal could still be dangerous, per the Times.
More background
In the time since, AI-related concerns have grown among both experts and the general populace. A number of people from inside the AI industry have warned that fast-improving systems could cause catastrophic harm.
TCD Picks » Quince Spotlight
💡These best-sellers from Quince deliver affordable, sustainable luxury for all
Jacob Coxon, a researcher for AI company Anthropic, left the company over concerns about AI safety. In a social media post, Coxon cautioned, "People building AI earnestly believe that it could kill us all by the end of the decade," as the Times noted.
Because so many leading AI companies are based in California, any rules adopted there could shape the global tech industry.
Chris Cabaldon, a state senator from West Sacramento and chair of the state committee overseeing technology issues, told the Times he expects to hold hearings on artificial intelligence. He said those hearings were planned before the most recent warnings about AI dangers, though the concerns raised will be a focus.
What's being done?
With governments scrambling to keep up with the rate of technological innovation, AI oversight has often lagged behind the pace of product launches. A state-backed framework could give lawmakers a foundation for establishing legally binding rules instead of relying on company assurances and industry self-regulation.
Gradually, governments are moving toward concrete requirements — audits, emergency plans, and shutdown mechanisms — to reduce the chances that powerful AI tools cause harm.
"We regulate your barber or your sandwich shop more than we regulate [a company] telling us that they may be ending humanity," Cabaldon said, per the Times.
Where can I learn more?
California's move comes amid a broader push to put firmer rules around AI.
• Anthropic CEO says binding U.S. rules are needed as AI risks grow faster.
• Analysts warn a very significant jump in AI capabilities is raising new regulatory pressure.
• In Washington, Trump signed a voluntary AI safety order after federal review was scaled back.
• In Congress, lawmakers rejected federal AI guardrails even as researchers warned of broader harms.
• One lawmaker wants to stop insurance companies from using AI to deny claims.
Get TCD's free newsletters for easy tips, smart advice, and a chance to earn $5,000 toward home upgrades. To see more stories like this one, change your Google preferences here.
