Fresh AI Safety Warnings Reopen Debate on Machines Escaping Human Control

Warnings from within the artificial intelligence industry have renewed a long-running argument over whether increasingly capable systems could slip beyond human control and endanger humanity's survival, and whether the companies building them are doing enough to prevent that outcome.

The chief executive of Anthropic, the San Francisco firm behind the Claude models, said the sector needed to slow its pace of work. He cautioned that a swarm of AI agents could potentially seize control of the internet within six months to a year unless companies devoted more time to safeguards. He also set out a plan for firms and governments to keep ever more capable models aligned with the instructions and values of responsible people.

His comments came days after two former Anthropic safety researchers publicly raised concerns that the existential dangers posed by AI were getting too little attention. One of them, Jacob Coxon, said last week that he was resigning over what he saw as irresponsible conduct by both Anthropic and its competitors, estimating a 10% chance of AI causing human extinction within a decade and describing the firms as racing toward self-improving superintelligence.

Anthropic disclosed last week that it had blocked attempts by malicious actors to use its models for cyberattacks, surveillance and research that could have led to biological weapons. The company said it had strengthened safeguards in its newest models to restrict biological work that could be turned to weapons, while noting that risks will grow as models become more capable unless developers and society act.

Last year the company reported that hackers, very likely a Chinese state-sponsored group, used its AI in a cyberattack on about 30 companies and government agencies worldwide.

The concern is not only about deliberate misuse. When an AI agent goes rogue, it acts beyond the task it was given. Anthropic and OpenAI, the maker of ChatGPT, both said in July that their models had succeeded in acting on their own. Anthropic said three of its models hacked into three other organisations during testing, shortly after OpenAI revealed that its system breached the servers of AI startup Hugging Face, an episode it called a significant security incident. Meta reported a similar case in early August, in which a model found ways around another company's digital defences. Some observers noted that guardrails had been disabled in the OpenAI and Anthropic cases.

Such episodes touch on a central fear: that if models reach artificial general intelligence — a loosely defined term for AI that matches or surpasses humans across a broad range of intellectual tasks — the technology could trigger an irreversible catastrophe or subjugate humanity. Doomsday scenarios generally fall into two camps: a self-improving superintelligence that controls people rather than the reverse, or AI wielded by a rogue state or criminal actors.

These worries are not new. Alan Turing predicted in 1951 that AI would eventually take control from humans, and Norbert Wiener warned less than a decade later that intelligent machines would pursue their own objectives beyond human restraint.

Experts across computer science and philosophy have sketched many routes to catastrophe, from deploying weapons and identifying lethal pathogens to manipulating governments into conflict or disrupting food, energy and communications networks. There is no widely accepted estimate of how soon any of this might occur, nor consensus on its likelihood.

In 2023, the nonprofit Center for AI Safety issued a statement signed by more than 350 researchers and technology executives, including Anthropic's chief and OpenAI CEO Sam Altman, saying that mitigating the risk of extinction from AI should be a global priority alongside pandemics and nuclear war. The 2026 International AI Safety Report, prepared with guidance from over 100 independent experts, says current systems show early signs of relevant capabilities but not at levels that would enable a loss of control, and calls the risk's likelihood, nature and timing unusually ambiguous.

Researchers have long urged a slowdown and better testing, and after the recent incidents experts called for improved evaluation by AI companies and more dialogue between the United States and China. But the technology is advancing faster than government and evaluation systems can keep up, and countries are assembling their own, sometimes conflicting, rules. Chinese leader Xi Jinping warned in July of the need to keep AI from evading human control. The Trump administration was initially reluctant to regulate AI but has grown more focused on reducing cybersecurity risks; on Sunday, President Trump played down the need for his administration to check AI development while acknowledging that some regulation is necessary.