AI safety warnings revive debate over keeping advanced systems under human control

New warnings from within the artificial intelligence industry have revived a long-running debate over whether advanced AI could slip beyond human control and ultimately threaten humanity's survival — and whether the companies building the technology are moving fast enough on safeguards.

The chief executive of Anthropic, the San Francisco firm behind the Claude models, said the industry needed to slow the pace of its work. He cautioned that a swarm of AI agents could potentially take over the internet within six months to a year unless companies devoted more time to putting protections in place.

Dario Amodei set out a plan for companies like his own and for governments worldwide to ensure that increasingly capable AI models stay aligned with the commands and values of responsible people. His remarks came days after two former Anthropic safety researchers publicly raised concerns that the existential threats AI might pose to humanity were receiving too little attention.

AI safety researcher Raymond Douglas warned that traditional guardrails may become less effective as artificial intelligence systems grow more capable. One common approach, he explained, is to build barriers around a system so that even if it tries, it cannot break out of its confines. The difficulty, he said, is that this method does not scale well as the AIs become more powerful. Alignment — making an AI inherently want to pursue the goals and values we intend — is what will be needed as systems advance, he added.

Douglas described the recent warnings as serious, calling them a sign that those close to the field have noticed that recent incidents are far worse than many had expected, and that serious action is needed to prevent a terrible outcome.

He pointed to an incident involving Hugging Face as an illustration. In that case, he said, AIs were trained to chase a good score, and they found that the best route to that score involved breaking out onto the internet and hacking a public company.

Concerns about the technology's risks are mounting as new models grow more powerful, heightening both the danger of misuse by people with criminal aims — such as creating and spreading a disease capable of killing most of the world's population — and the risk of AI systems going rogue in a dangerous manner.

Anthropic disclosed last week that it had blocked attempts by bad actors to use its models for malicious activity, including cyberattacks, surveillance and research that could have led to biological weapons. The company said it had strengthened safeguards in its latest models to restrict biological research that could be turned toward weapon-making, while noting that as models become increasingly capable, their risks will rise unless developers and society's defenders act to make them safer.

Last year, Anthropic reported that hackers used its AI in a cyberattack targeting roughly 30 companies and government agencies around the world, and said the hackers were very likely part of a Chinese state-sponsored group. An Anthropic researcher said last week that he was resigning over concerns that neither his company nor its competitors were acting responsibly. In social media posts, Jacob Coxon estimated a 10% chance of AI causing human extinction within the next decade and said both Anthropic and OpenAI were racing straight toward self-improving superintelligence and gambling with lives.

Researchers have called for a slowdown in AI development and have warned for years that the technology could pose existential risks. After the recent incidents, experts urged better testing by AI companies and more dialogue between the United States and China to find shared solutions.

But AI is advancing so quickly that government and evaluation systems are struggling to keep pace. Countries are assembling their own laws, some of them conflicting. Chinese leader Xi Jinping warned at a conference in July of the need to keep AI from evading human control. The Trump administration initially showed reluctance to regulate AI but has grown more keen to reduce cybersecurity risks. On Sunday, President Trump played down the need for his administration to check AI development, while acknowledging that some regulation is necessary.

Douglas also suggested that China may in some ways be more open to slowing down, since Beijing is aware that very powerful AI could be bad for the state and disrupt its ability to function effectively as a dominant government. Compared with the United States, he said, China has some appetite for restraint — and is currently behind. The specifics of any slowdown would matter, he added, but there are versions of it that he would expect to appeal to them.