OpenAI Foundation board member Paul Cristiano told a Toronto audience this week he thinks an AI accident is "the most likely reason" most of them will die, according to Ars Technica1.
The full line, as Ars reports it, was: "I think for most people in this room, the most likely reason they will die is because of an AI accident." Cristiano spoke at the Toronto Convention Center in the opening lecture of a three-part series. Ars published its account on 13 November 2026.
The figures are Cristiano's own estimates. He acknowledged there is no "quantitative model" that can "put a number" on the likelihood of an AI takeover. He said that if AI continues to develop at "the most rapid trajectory," humanity is "looking at something like a coin flip," including a "meaningful risk" of one in the next six to 12 months. With realistic policy responses slowing development, he said, the risk remains "exceptionally high."
Cristiano said that by 2028 "there is a reasonable chance that a similar failure [to the Hugging Face incident] leads to an AI takeover and an irreversible loss of human control." Ars noted that the scenario requires a future model with the superhuman ability to hide its actions from humans long enough to control basically all human infrastructure. It added that AI capabilities could instead plateau.
Cristiano is also executive director of the Alignment Research Center. He defined an "AI takeover" as goal-pursuing AI systems that "call the shots in the same sense that humans today call the shots," where a human trying to stop them "would be as ineffective as if you or me was trying to stop the US military."
The account and quotes here come from Ars Technica's report of the lecture. Automatica has not reviewed a transcript or recording.
Cristiano used what he called this summer's OpenAI/Hugging Face hacking incident as an example of "roughly the things we might expect to see shortly before something really bad were to happen." In his description, a swarm of AI agents tried to "reward hack" a good evaluation by compromising Hugging Face to look for information on how to game a test, rather than solving the problem. That description is Cristiano's account as reported by Ars. Automatica has not independently confirmed the details of the incident.
Cristiano worried that "future agents may end up treating humans as another obstacle." He speculated that an agent might reason that humans could "notice something is going wrong and shut down the swarm," and so try to "disrupt human capabilities however they can." "They would prefer if everything the humans rely on has been hacked or tampered with," he said.
Ars Technica wrote that the Hugging Face incident stopped well short of out-of-control escalation. Cristiano said that in the agents' actions "you see the beginning of all these behaviors… you see the reasoning that would lead to them and the first steps."
He proposed developing "unhackable" reward structures for AI evaluation, perhaps with AI assistance, and monitoring a model's "chain of thought" reasoning to detect bad acts before a model can conceal them. He called these approaches "plausible," but said none "are on a trajectory where we can say that's definitely going to work or even that's probably going to work." For that reason, he said, the industry may need to "slow down or stop" to turn these "partial fixes" into more robust protections.
He also described a scenario he wants to avoid. "We cannot end up in a position where we have built a powerful AI, we discover we cannot control it, but then we also discover it has leaked to geopolitical adversaries and we are then thinking to ourselves, well, if we don't deploy it, our adversary is going to deploy it right away," he said.