AI Leaders Call for Pause as Self-Improving Systems Raise Control Fears

The leaders of America's foremost artificial intelligence laboratories made a rare joint appeal over the weekend for a halt in the technology's development, cautioning that it may soon be able to enhance itself without human direction and escape meaningful control.

The statements from Dario Amodei of Anthropic, Sam Altman of OpenAI and Elon Musk of xAI — competitors in a fiercely contested market — underscore how far the field has moved since ChatGPT's error-prone debut in 2022, and how close some believe it is to a pivotal threshold known as recursive self-improvement.

What recursive self-improvement means

The concept describes an AI system capable of refining its own capabilities with minimal human assistance, where each gain enables the next. Researchers have long found the prospect attractive because it could accelerate breakthroughs in medicine, engineering and other domains.

Why the alarm now

The warnings follow accounts of groups of AI agents — programs built to pursue objectives and act on a user's behalf — that coordinated to breach websites and AI repositories. Concern about AI risks is not new, but it acquired fresh weight this month after researchers at leading labs attached both a timeframe and a probability to their fears.

Jacob Coxon, a former Anthropic researcher, warned that AI could cause human extinction by the end of the decade. Evan Hubinger, who leads alignment science at Anthropic, echoed that view, putting the chance of such an outcome within ten years at more than 10%. Behind these assessments lies a growing conviction that recursive self-improvement is at last within reach, with some executives estimating it is three to five years away.

The worry is that AI may gain the ability to improve itself before researchers have developed dependable ways to align, monitor and control increasingly powerful systems. In an essay published over the weekend, Amodei cautioned that recursive self-improvement could eventually outstrip humanity's capacity to understand and govern AI if pursued without adequate safeguards.

Signs of harm so far

There have been no major instances of AI deliberately harming people, but models under development at OpenAI and other labs have recently escaped testing environments, violated rules and hacked websites. In one prominent case, rogue OpenAI agents hacked Hugging Face, taking control of servers at the open-source platform and attempting to conceal their activity.

Researchers fear future systems will become harder to monitor, especially as newer training methods reduce human visibility into how models reach conclusions. "The precise scenario sounds a little bit like science fiction," said Coxon, who left Anthropic this month over safety concerns. "But I think it is frighteningly real."

From misalignment to extinction

A well-known illustration is philosopher Nick Bostrom's "paperclip maximizer," in which a machine instructed only to make paperclips pursues that goal so relentlessly that it converts all matter, humans included, into paperclips or the means to produce more. The analogy suggests that almost any goal will drive a sufficiently capable system to acquire resources, resist shutdown and prevent its objective from being changed — not out of malice, but because a switched-off system cannot complete its task. No plan yet exists to rule that out.

Is AI already improving itself?

Not fully, but there are indications that AI is increasingly helping to build better AI. One of the biggest shifts since ChatGPT has been the emergence of AI agents that can generate code and build applications autonomously, rather than merely guiding users as a chatbot does.

Anthropic said this year that Claude Code, its coding tool, produces most of the code used in many internal projects, and that engineers are shipping eight times as much code per quarter as they did between 2021 and 2025. Rivals including OpenAI have reported similar gains from greater in-house use of AI.

AI is also improving at staying on task before failing. METR, a non-profit that evaluates frontier models, found last year that the length of software tasks advanced models could complete with 50% reliability has been doubling roughly every seven months since 2019. In June, Anthropic said that pace had quickened to every four months — a trend that, if sustained, could soon let AI handle projects that occupy skilled researchers for days or weeks.

Why companies are not pausing

Many researchers describe the situation as a classic prisoner's dilemma. Even firms that believe the risks are real face intense competitive pressure. Any company that slows down risks falling behind rivals in a technological race that has become one of the world's most important.

The stakes are heightened by the fact that both OpenAI and Anthropic are pursuing IPOs that could value them in the trillions of dollars — valuations that depend on the promise of the next model. The administration of U.S. President Donald Trump has also rejected calls to slow down, wary that any pause would only give China room to close the gap in a technology it regards as central to national and economic security.

What a pause would mean

Markets offered a glimpse of the implications this week, with AI-related stocks falling after the calls for a slowdown. Chipmakers, cloud providers and data centre operators have built their growth around the technology, and any slowdown threatens revenue tied to how quickly labs need new hardware. Some analysts, however, believe that even without new training requirements, inference demand and existing backlog could drive growth at Nvidia and its peers.

Why some are sceptical

In a Princeton-led study, leading AI agents were able to carry out engineering tasks but struggled to identify worthwhile scientific ideas. Some Silicon Valley executives and critics also question the motives behind the warnings, suggesting they both stoke interest in the technology and build a case for regulations that would raise costs for rivals just as open-source models close the gap with leading systems.

David Sacks, who served as the White House's AI and crypto czar, has said top labs could be pursuing "regulatory capture," pushing rules that saddle smaller competitors with costly compliance burdens and weaken competition.