Anthropic urges global AI pause as human control narrows
The San Francisco-based firm calls for verifiable coordination between the US and China, citing internal data on accelerating development loops, despite political and industry scepticism.
Anthropic, the developer of the Claude family of artificial intelligence models, has issued a report calling for a global pause on the development of the most powerful AI systems. The company warns that cutting-edge models are showing signs of becoming increasingly difficult for humans to control, potentially leading to a narrowing of the human role in the development process. Anthropic suggests that a worldwide slowdown would "likely be a good thing" to allow societal structures and alignment research to keep pace with technological advances.
The firm argues that a pause requires global coordination, specifically involving major AI companies and governments in the US and China, under verifiable rules. Without such a mechanism, Anthropic stated that companies and governments would be forced to make difficult safety decisions while under intense competitive and geopolitical pressures. The company compared the challenge of AI regulation to nuclear arms control treaties but noted it is more difficult because AI training is easier to conceal than physical infrastructure like missile silos.
Internal data highlighted by Anthropic indicates that AI is accelerating its own development, creating a feedback loop that could lead to "recursive self-improvement," where systems teach themselves to get smarter with minimal human intervention. The report cautioned that while recursive self-improvement is not inevitable, the evidence suggests the human role is narrowing at each step in the AI development process, potentially arriving sooner than most governments are prepared for.
The proposal faces significant headwinds in Washington and Silicon Valley, where critics argue that Anthropic’s focus on worst-case scenarios overstates the risks and serves as a strategy to slow down rivals under the guise of safety. There is substantial concern that any slowdown in AI development could cede a decisive strategic edge to China in what is viewed as the defining technology race of the century. Despite this, the White House has acknowledged the capabilities of Anthropic’s "Mythos" model, which is currently deployed only to a small number of vetted organisations and is not available to the general public due to its cybersecurity capabilities.
In response to the growing complexity of the sector, US President Donald Trump stated he discussed cooperating with China on AI safety issues during a recent visit to Beijing. Trump also signed an executive order this week mandating a 30-day preliminary government review of the most powerful US AI models before their release. Anthropic plans to convene government officials, scientists, advocacy groups, and competing AI firms in the coming months to design a coordination mechanism.
