There’s a popular sci-fi cliché that one day artificial intelligence goes rogue and kills every human, wiping out the species. Could this truly happen? In real-world surveys AI researchers say that they see human extinction as a plausible outcome of AI development. In 2024 hundreds of these researchers signed a statement that read: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
Okay, guys.
Pandemics and nuclear war are real, tangible concerns, more so than AI doom, at least to me, a scientist at the RAND Corporation. We do all kinds of research on national security issues and might be best known for our role in developing strategies for preventing nuclear catastrophe during the cold war. RAND takes big threats to humanity seriously, so I, skeptical about AI’s human extinction potential, proposed a project to research whether it could.
On supporting science journalism
If you’re enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
My team’s hypothesis was this: No scenario can be described where AI is conclusively an extinction threat to humanity. In other words, our starting hypothesis was that humans were too adaptable, too plentiful and too dispersed across the planet for AI to wipe us out using any tools hypothetically at its disposal. If we could prove this hypothesis wrong, it would mean that AI might be a real extinction threat to humanity.
Many people are assessing catastrophic risks from AI. In the most extreme cases, some people assert that AI will become a superintelligence, with a near certain chance that it will use novel, advanced tech like nanotechnology to take over and wipe us out. Forecasters have predicted the likelihood of existential risk from an AI catastrophe, often arriving between a 0 and 10 percent chance that AI causes humanity’s extinction by 2100. We were skeptical of the value of predictions like these for policymaking and risk reduction.
Our team consisted of a scientist, an engineer and a mathematician. We swallowed any of our AI skepticism and—in very RAND-like fashion—set about detailing how AI could actually cause human extinction. A simple global catastrophe or societal collapse was not enough for us. We were trying to take the risk of extinction seriously, which meant that we were interested only in a complete wipeout of our species. We also weren’t interested in whether AI would try to kill us—only in whether it could succeed.
It was a morbid task. We went about it by analyzing exactly how AI might exploit three major threats commonly perceived to be existential risks: nuclear war, biological pathogens and climate change.
It turns out it is very hard—though not completely out of the realm of possibility—for AI to kill us all.
The good news, if I can call it that, is that we don’t think AI could kill us all with nuclear weapons. Even if AI somehow acquired the ability to launch all of the 12,000-plus warheads in the nine-country global nuclear stockpile, the explosions, radioactive fallout and resulting nuclear winter still would likely fall short of an extinction-level event. Humans are far too plentiful and dispersed for the detonations to directly kill all of us. AI could detonate weapons over all the most fuel-dense areas on the planet and still fail to produce as much ash as the meteor that likely wiped out the dinosaurs. There are also not enough nuclear warheads on the planet to fully irradiate all the planet’s usable agricultural land. In other words, an AI-initiated nuclear Armageddon would be cataclysmic, but it would likely still fall short of killing every human being, because some humans would survive and have the potential to reconstitute the species.
On the other hand, we deemed pandemics to be a plausible extinction threat. Previous natural plagues have been catastrophic, but human societies have survived and soldiered on. Even a minimal population (likely a few thousand members) could eventually reconstitute the species. A hypothetically 99.99 percent lethal pathogen would leave more than 800,000 humans alive.
We determined, however, that a combination of pathogens could likely be designed to achieve nearly 100 percent lethality, and AI could be used to deploy such pathogens in a manner that assured rapid, global reach. The key limitation is that AI would need to somehow infect or otherwise exterminate communities that would inevitably isolate themselves when faced with a species-ending pandemic.
Finally, if AI were to accelerate garden-variety anthropogenic climate change, it would still not rise to an extinction threat to all of humanity. Humans would likely find new environmental niches in which to survive, even if it involved moving to the Earth’s poles. To make Earth completely uninhabitable for humans would require AI pumping something much more potent than carbon dioxide into the atmosphere. That’s the good news.
The bad news is that those much more powerful greenhouse gases exist. They can be produced at industrial scales. And they persist in the atmosphere for hundreds or thousands of years. If AI were to evade international monitoring and orchestrate the production of a few hundred megatons of these chemicals (that is less than the mass of plastic that humans produce each year), it would be sufficient to cook the Earth to the point that there is no environmental niche left for humanity.
I do want to make this clear: None of our AI-extinction scenarios could happen by accident. Each would be immensely challenging to carry out. AI would somehow have to overcome major constraints.
In the course of our analysis, we also identified four things that our hypothetical super-evil AI has to have: First, it would need to somehow set an objective to cause extinction. AI also would have to gain control over the key physical systems that create the threat, like nuclear weapon launch control or chemical manufacturing infrastructure. It would need the ability to persuade humans to help and hide its actions long enough to succeed. And it has to be able to survive without humans around to support it, because even once society started to collapse, follow-up actions would be required to cause full extinction.
If AI did not possess all four of these capabilities, our team concluded its extinction project would fail. That said, it is plausible to create AI that has all of these capabilities, even if unintentionally. Moreover, humans might create AI with all four of these capabilities intentionally. Developers are already trying to create agentic, or more autonomous, AI, and they’ve already observed AI that has the capacity for scheming and deception.
But if extinction is a plausible outcome of AI development, doesn’t that mean we should follow the precautionary principle? That is to say: Shut it all down because better off safe than sorry? We say the answer is no. The shut-it-down approach is only appropriate if we don’t care much about the benefits of AI. For better or worse, we care a great deal about the benefits AI will likely bring, and it is inappropriate to forgo them to avoid a potential but highly uncertain catastrophe, even one as consequential as human extinction.
So will AI one day kill us all? It is not absurd to say that it could. At the same time, our work also showed that humans don’t need AI’s help to destroy ourselves. One surefire way to lessen extinction risk, whether or not it stems from AI, is to increase our chances of survival by reducing the number of nuclear weapons, restricting globe-heating chemicals and improving pandemic surveillance. It also makes sense to invest in AI safety research, whether or not you buy the argument that AI is a potential extinction risk. The same responsible AI development approaches that mitigate risk from extinction will also mitigate risks from other AI-related harms that are less consequential, and also less uncertain, than existential risks.
This is an opinion and analysis article, and the views expressed by the author or authors are not necessarily those of Scientific American.
