We Need an International Treaty to Ban Superintelligence
No country has an interest in building AI that could wipe out humanity.
Want to join a Zoom call with Francis Fukuyama to discuss “The End of History”? Of course you do! Frank is joining our Intellectual Bootcamp this week, Friday June 26 at 12 noon. You must be a paying subscriber to participate in the bootcamp—so click here to take out a paid subscription today! Once you have subscribed, please email events@persuasion.community to be added to the mailing list and receive the Zoom link.
As today’s AI companies race to create artificial intelligence that is smarter than humans, governments around the world are failing to meet the moment.
Just this April, Anthropic withheld its Mythos model from wide release due to its unprecedented cyberattack capabilities. General Joshua Rudd, head of the National Security Agency and Cyber Command, confirmed that Mythos “broke into almost all of our classified systems, not in weeks, but in hours.” This led the Trump Administration to issue an executive order setting up voluntary pre-deployment reviews of AIs. When Anthropic deployed a scaled-back version of Mythos called Fable 5 to the public, the Trump administration used export controls to bar foreign nationals from using the models, forcing Anthropic to disable both Fable 5 and Mythos.
It’s clear that the U.S. government is starting to take the national security implications of powerful AI seriously. But it is still missing the bigger picture. Pre-deployment evaluations and export controls fall far short of addressing the threats from increasingly capable AIs, because the threats we’re facing aren’t just shockingly capable cyberweapons. It’s much worse than that. The leading AI companies are explicitly trying to build superintelligent AI, or “superintelligence.” Such AIs would be vastly more capable than humans, fully autonomous, and able to overpower countries’ national security institutions.
The entire international security architecture is built around the assumption that the most dangerous weapons are controlled by humans, and that the decision to take lives ultimately rests with humans. Superintelligence would upend this. Unlike nuclear weapons—the most lethal weapons we have today—superintelligent systems would not be tools humans can leverage, but rather agents able to pursue their own objectives with no human control over their actions. The leading AI companies are actively worsening the problem by making AIs as autonomous as possible and cutting humans out of the process of AI development.
Superintelligence thus represents the ultimate principal-agent problem, and it’s nowhere near being solved. Most alarming of all, Nobel Prize winners, leading AI scientists, and even the CEOs of the top AI companies warn that superintelligence poses an extinction risk to humanity. OpenAI’s CEO Sam Altman has repeatedly said that superhuman machine intelligence is the greatest threat to humanity’s continued existence, yet he affirms that OpenAI is “before anything else … a superintelligence research company.”
As a handful of companies accelerate toward superintelligence, is it already too late to step back from the brink? No. But the window to act is narrow, and it is closing.
The only solution is an international prohibition on the development of superintelligent AI. Governments should pursue this prohibition urgently, because there is no technical solution in sight that would allow humanity to remain in control over AIs vastly smarter and more capable than us.
Some argue we can solve this through “alignment” research that would let us create AIs that do what humanity wants. But the astonishing growth in AI capabilities is drastically outpacing research into understanding and controlling AIs. To see why that gap matters, consider how the systems are built.
Right now, engineers essentially grow AI systems out of enormous piles of data, instead of coding the systems line by line like traditional software. They use intense amounts of computational power in large data centers to train these systems. What emerges on the other end of training are AI systems whose internal workings are opaque even to their developers.
While engineers can test what AI is good at and try to slap some guardrails around it, they do not know how the systems actually work. Dario Amodei, the CEO of AI company Anthropic, has said we understand maybe three percent of what is happening inside AI models.
The inherent unpredictability and uncontrollability of these systems presents massive risks as we get closer to superintelligent systems. And even if one brilliant scientist found a solution, without any government intervention there is no one to stop a different company, or country, from rushing ahead with superintelligent AI that can kill everyone.
Fortunately, it would be possible to both monitor and enforce an international agreement prohibiting superintelligence, because there is a large industrial process behind advanced AI systems. This is akin to the extensive, largely traceable process required to create nuclear weapons. The supply chain is unusually narrow and controlled by Western countries and like-minded democracies. The world’s most advanced AI chips are fabricated almost exclusively by TSMC in Taiwan. The lithography machines required to make those chips are built by a single company, ASML, in the Netherlands. And cutting-edge chip designs come overwhelmingly from one American firm, NVIDIA.
Additionally, the data centers training and continuously running powerful AIs use power on the scale of small cities. Their footprints are easily tracked using satellite imagery. The Federation of American Scientists recently demonstrated this by mapping the buildout of a large-scale AI data center in the United Arab Emirates from 2024 to 2025, using freely available imagery. The size of the industrial process is precisely what makes monitoring, and potential intervention, possible.
But the window to act is closing. In the coming decade, we can expect algorithms for AI to improve, while hardware like advanced chips could get smaller and more efficient. It’s possible that within a few years, the resources required to build superintelligence could become small enough that monitoring is rendered impossible. This could also eventually make smaller actors, including terrorists and other non-state groups, capable of building such systems.
We need to act before that happens.
The United States should domestically prohibit superintelligence and set a policy to prevent its development globally. It should work with other countries to establish an international agreement prohibiting the development of superintelligence using a “trust but verify” regime, similar to how we have handled nuclear deterrence. While such an agreement is a tall order amid today’s tumultuous geopolitics, it is well within the reach of existing tools of statecraft.
Countries party to the agreement would ensure that superintelligence is not domestically developed. If anyone outside the group tries to develop it, countries in the group could respond to deter that development. They could use export controls, sanctions, or intelligence cooperation. No country has an interest in any actor, including itself, developing a technology that could overpower the security forces of any government and threaten humanity with extinction.
None of this will be easy. The ongoing great power competition with China is tense, and there is minimal appetite in Washington for international cooperation. But the same was said about nuclear nonproliferation, and there has been no nuclear detonation in conflict since 1945.
The political coalition for prohibition is already broader than many realize. In the fall of 2025, leading AI scientists and political figures across the spectrum, like former U.S. National Security Advisor Susan Rice, President Trump’s former adviser Steve Bannon, and former U.S. Chairman of the Joint Chiefs of Staff Admiral Mike Mullen, called for a prohibition on the development of superintelligence. Meanwhile, a coalition of more than 150 lawmakers across the UK and Canada are warning about the extinction-level threat from superintelligence and calling for binding limits, up to a ban on developing such technology.
Yet most policymakers still aren’t taking the danger seriously. They’ve only heard the nice stories of sky-high profitability sold to them by AI executives. What’s missing is both the widespread recognition that superintelligence is an unprecedented national security threat, and the political will to act. The right solution is a doctrine of superintelligence prevention that protects the United States and the rest of the world.
With it, we can still avert the catastrophe we’re racing toward.
Andrea Miotti is founder and CEO of ControlAI, a non-profit organization working to keep humanity in control of advanced AI.
Follow Persuasion on X, Instagram, LinkedIn, and YouTube to keep up with our latest articles, podcasts, and events, as well as updates from excellent writers across our network.
And, to receive pieces like this in your inbox and support our work, subscribe below:
 | A guest post by
|
A fine piece on desirable governance. One suspects that at this point a much needed AI arms control treaty will come to pass only if the collective super AIs decide among themselves to allow it!
The situation is worse than the author states because these 'superintelligent AIs' are not free to make up their own mind but are controlled by humans, could be evil ones.
Humans must provide the will to act by giving commands e.g. "kill them all I don't care how you do it".
If a 'self driving car' veers off the road into a crowd of people we might anthropomorphise the car and say 'the car did this' but of course the event is actually the result of many human decisions from software development to the actual driver being too slow to take over to politicians who allow lax rules for these vehicles due to lobby pressure. This diffusion of responsibility is the real problem here, if everyone is responsible then no one is.
Suppose a superintelligent AI is stealing money from your bank account. "The superintelligent AI is doing it by its own volition" cry the AI boosters so nothing can be done! Of course in reality the cash is going into someone's account who set up the AI to do it.
In addition the creation of AIs with evil intent may not require munching through the data of the entire internet and so may not require the most massive facilities.
The self interested witterings of dodgy characters such as Sam Altman are not a useful guide to anything.