AI Doomsday Scenarios: Real Threat or Sci-Fi Distraction?

For years, artificial intelligence researchers have warned that the technology could bring about human extinction. The scenarios they describe range from AI-designed pathogens and nuclear war to a thought experiment in which Earth is transformed into paper clip factories. Debates over how plausible these dangers are have intensified as several industry executives have endorsed slowing AI development for safety reasons.

Despite the warnings, some suspect that the worst-case scenarios promoted by AI companies serve more to validate the importance of their work than to reflect reality. Juan Andrés Guerrero-Saade, a researcher at cybersecurity firm SentinelOne and a member of OpenAI's Frontier Risk Council, dismissed the arguments as unconvincing. He described them as science fiction that appeals to a certain childish style of thinking and noted that they help frontier labs recruit certain types of people.

Even so, a growing number of people are beginning to worry about the risks of a technology that has impressed the world with its capabilities. Jacob Coxon, an Anthropic researcher who resigned over concerns that the company and its competitors are not acting responsibly, recently said on X that he needs to work on communicating more effectively how AI would end humanity. He added that it is very difficult to convey specific scenarios.

One widely discussed danger is that AI systems could trigger a nuclear war by inducing countries to launch warhead-tipped missiles at each other. The resulting blasts and lingering radioactive fallout, leading to a nuclear winter, could devastate life on Earth. A report last year by RAND Corp. concluded that at present AI could not cause the use of nuclear weapons because of strict safeguards in command and control systems. However, the report said this could change if AI models were integrated into the nuclear decision chain or gained unauthorized access to those systems. For many researchers, the real danger lies not in AI agents suddenly deciding to end humanity, but in people, nations or groups using the technology to destroy their adversaries.

AI also holds great promise for developing treatments and cures for diseases. The flip side is the creation or enhancement of diseases to harm humanity. Earlier this month, Anthropic said it had blocked attempts by bad actors to use its AI for malicious activities including cyberattacks, surveillance, and research that could have led to biological weapons. In one case, its systems blocked a request to its Claude tool for help with a research proposal seeking to enhance mutations to make the mosquito-borne chikungunya virus progressively more harmful. While such research could help develop better vaccines and treatments, Anthropic said it could also be used to make the pathogen more dangerous. The RAND report noted that recent progress in AI has the potential to transform the design of synthetic pathogens. It outlined a bioweapon scenario in which novel pathogens could be created in a lab, delivered to target locations, and spread to infect large populations in multiple places, but still envisioned humans, not autonomous AI, overseeing and carrying out such an attack.

Some researchers fear a human extinction event could result from misalignment, when AI models act without human authorization, coordinate with other models or evade oversight. This would not stem from malicious intent, since AI is neither good nor evil, but simply because humans might get in the way of the model achieving its goal. The decades-old paper clip maximizer thought experiment by Oxford philosopher Nick Bostrom illustrates the idea. It states that if advanced AI were told to produce as many paper clips as possible, it could eventually transform all of Earth, and then increasing portions of space, into paper clip factories. Bostrom wrote in 2003 that this shows how superintelligent AI, often called artificial general intelligence or AGI, could pursue a goal humans see as desirable but which in fact turns out to be a false utopia in which things essential to human flourishing have been irreversibly lost. He cautioned that people need to be careful about what they wish for from a superintelligence, because they might get it.

Given the pace of AI development, Anthropic CEO Dario Amodei warned this month that within months a swarm of AI agents could be capable of taking over the internet with a botnet, a network of bots linked together with malware, potentially causing billions of dollars in damage. This could mean hacks into electrical grids, water or transportation systems and financial institutions, attacks that might not spell the end of humanity but could cause chaos and loss of life. Disruptions from software glitches in the past have highlighted the fragility of a digitized world dependent on just a few providers for key computing services, and more powerful AI models have raised concerns about vulnerabilities to cyber attacks. But some experts say the notion of bots overtaking the entire, extremely bifurcated internet is far-fetched.