Some of the field's most influential AI researchers and lab leaders say governments should prepare for the possibility that AI systems could become capable enough to design better AI. That would allow artificial intelligence to move faster than humans can effectively monitor it.
Here's what to know
As The Guardian reported, a paper warns that automated AI research could become good enough to speed up its own improvement and trigger an AI "intelligence explosion." The report was co-authored by more than 20 people, including Nobel laureate and AI pioneer Geoffrey Hinton, Canadian computer scientist and AI pioneer Yoshua Bengio, Anthropic co-founder Jack Clark, and OpenAI chief scientist Jakub Pachocki.
Their concern centers on AI systems that can make advances without human intervention. The paper grouped that recursive self-improvement under the label of automated AI research and development.
According to the authors, once AI can carry out research and development at an expert level, one developer could effectively direct work comparable to that of millions of top human researchers.
That could bring enormous benefits, including faster medical discoveries and other major technological breakthroughs. But the report warns that the same dynamic could also weaken human control.
"An intelligence explosion could be the most consequential technological development in human history, compressing years of progress into months or less, threatening human control over AI systems, and severely eroding checks on power within and between states, companies, and branches of government," the paper states, per The Guardian.
TCD Picks » Quince Spotlight
💡These best-sellers from Quince deliver affordable, sustainable luxury for all
More background
Some AI companies are using AI to build their own systems. Anthropic says AI generates 80% of its own code, while OpenAI uses autonomous AI agents for tasks including training new models.
"Preliminary evidence suggests that a software-driven intelligence explosion is possible," the report states.
At the same time, the authors stress that the consequences are uncertain, not inevitable. Supply chains, regulations, and deployment bottlenecks could slow breakthroughs. AI could also accelerate safety work and risk mitigation.
Still, the researchers say the possible dangers are significant enough that governments at the highest levels should prepare.
The paper cites risks including cyber and biological threats advancing faster than defenses, states turning a modest lead in areas such as cyberspace into a decisive advantage, and humans losing the ability to steer critical systems as AI takes on more of the work.
What's being done?
The report lays out several policy ideas for governments. Among them are requiring transparent progress reports on AI R&D, placing independent auditors inside companies, capping how quickly AI systems can improve, coordinating with data centers so AI projects can be paused if necessary, keeping automated AI R&D systems isolated so they cannot slip beyond human control, and creating emergency response plans for different scenarios.
Some outside scrutiny is on the way. Anthropic and OpenAI have both agreed to let independent evaluators assess their models after warning that AI development was reaching a critical point for safety.
The authors say AI could fully automate projects that take humans months to complete by 2028.
In their view, safeguards must be in place before an intelligence explosion begins, because "once an intelligence explosion begins, the window for action may close."
Where can I learn more?
These stories examine the fallout from AI's rapid expansion, from mounting pressure on oversight and regulation to the strain on communities and infrastructure.
• Public backlash is growing as jobs, water, and electricity come under pressure and security fears deepen.
• In Congress, AI rules are being rejected even as climate and public health costs mount.
• New data show AI's explosive growth is driving energy costs and deepening inequality.
• Communities near expanding data centers are confronting environmental and water pressures tied to the AI boom.
Get TCD's free newsletters for easy tips, smart advice, and a chance to earn $5,000 toward home upgrades. To see more stories like this one, change your Google preferences here.
