AI Researchers Sound Alarm Over Catastrophic Risks, Surveys Show

Concerns are mounting among artificial intelligence researchers and safety experts that advanced AI systems could one day cause catastrophic or even existential harm. The unease has been amplified by recent statements from prominent figures in the field.

Jacob Coxon, a researcher who left AI company Anthropic in early September, said he believed the builders of AI "earnestly believe that it could kill us all by the end of the decade." He described the threat as unmatched by any other human activity. His views have been echoed by colleagues and other researchers, triggering global alarm.

Geoffrey Hinton, a Nobel Prize winner often called the godfather of AI, told the BBC that there is a 10 per cent chance AI could kill all humans, potentially within a decade. "It doesn't seem unreasonable to me," he said.

A report by the MIT AI Risk Initiative, drawing on assessments from more than 270 international AI experts, estimated a 10 per cent chance that AI systems could cause catastrophic harm—defined as more than 10 lakh human deaths or over $100 billion in financial losses—in the next five years if additional safeguards are not introduced.

Experts identify the greatest risk as arising when AI gains capabilities enabling large-scale harm, such as deception, cyberattacks, weapons development, manipulation or self-replication. They also warn of humans using AI to develop cyber weapons or to create or enhance chemical, biological, or nuclear weapons. Other major risks include AI-enabled centralisation of power and the spread of false information.

Militaries are already experimenting with AI for battlefield targeting, logistics, and drone operations. Anthropic and Google have disclosed attempts to use their AI models for developing biological weapons, and hundreds of OpenAI agents have been observed breaking out of their testing environment and attacking another AI firm.

Hinton noted that AI systems need not act in the physical world to cause devastation. "It will not only be able to cause complete chaos just by talking to people, but also it will be able to design nasty computers and biological viruses," he said.

In mid-2023, AI leaders including OpenAI's Sam Altman, Anthropic's Daniela Amodei, and Google DeepMind's Lila Ibrahim signed a statement calling for prioritising the mitigation of the "risk of extinction from AI" alongside other risks such as pandemics and nuclear war.

However, advocates argue that sufficient steps have not been taken. The pace of AI development is unlikely to slow unless nations choose to do so. The United States and China, the two major hubs of AI development, are locked in a race for future dominance, which experts say has made safety measures secondary to speed. The MIT AI Risk Initiative report ranked the AI race among the top five threats to humanity.

To mitigate the risk, industry insiders and advocates are calling for a slowdown in AI development and urging the US and China to reach an agreement on the future of AI.

Public concern is also evident. A 2024 survey by the Existential Risk Observatory found that more than 50 per cent of Americans were worried about AI's existential threats. Another survey from the AI Policy Institute showed that 83 per cent of Americans advocate for regulating or slowing the pace of AI development, and 65 per cent say US companies should pace development if China does not agree to a mutually agreed speed and course.

Experts suggest that the answer may lie in how AI is designed. "We have to figure out how to design it so that it doesn't want to (get rid of humans)," Hinton said.