The Misguided Panic About Superintelligence
You can’t ban something that doesn’t exist.
A recent Persuasion article by Andrea Miotti argues that “We Need an International Treaty to Ban Superintelligence.” His argument—like that of many others in what we might call doomer circles—is that large-scale Generative AI systems, like ChatGPT and Claude, are fast approaching a sharp transition from their current trajectory to runaway superintelligence, at which point they may escape human control, reprogram themselves to adopt new, self-directed goals, and possibly wipe out humanity entirely.
Everyone is familiar with this scenario because it’s a persistent trope of science fiction. We are constantly exposed to images of robots running amok, a staple of the modern entertainment ecosystem.
It’s certainly true that advanced AI deserves a thoughtful and appropriate regulatory framework. The benefits and risks of this powerful new technology are just coming into view, and developing the independent expertise to monitor, test, and approve at least the more powerful of these emerging systems should be a priority. This has been true of every major modern technical advance, from the printing press to railroads, automobiles, aviation, radio, TV, nuclear energy, recombinant DNA, and automated financial trading, to name just a few. The most successful approaches combine clear, government-mandated standards with administrative procedures that minimize the costs imposed on those working to market the relevant products and services.
Unfortunately, the doomers seriously misrepresent the actual nature and risks of advanced AI, and often propose solutions that are both unrealistic and ineffectual.
The basic framing is that AI is approaching and may soon exceed human intelligence, and that breaching this barrier represents a threat to the continued existence of the human race because the machines might—to use the science fiction trope—rise up, take over, and decide to wipe us all out. But this is not supported by any realistic understanding of the technology or its likely impact on society. Each of these assumptions is fatally flawed and deserves to be tempered if not completely debunked. Weaving them into a terrifying fantasy is both inappropriate and counterproductive, as it risks drowning out the more sober public deliberation that the technology deserves.
The core of the apocalyptic argument is easy to neuter. Simply substitute “save humanity” for “destroy humanity” and note how much more plausible it seems. A simple exercise of listing actual existential threats, and considering whether advanced AI may help us mitigate these risks, illustrates how much more likely the technology is to be useful than damaging.
To pick just a single example: It’s true that AI could be used to design highly contagious and deadly new viruses, and employing it toward this end deserves a clear international statement that this activity is not acceptable. (Though, it should be noted, this risk is hardly unique to AI, and is just as real without it.) But what about using the same AIs to rapidly develop countervailing vaccines, or even design a universal vaccine effective against all such threats, including natural ones? Fearing the former outcome, should we preclude the latter by banning the development of systems that might advance both goals?
The same is true of numerous clear and present existential threats, from the risk of a devastating asteroid strike to runaway climate change to mass starvation resulting from the collapse of global supply chains, electrical grids, or financial systems. Hampering AI development might inadvertently block us from precisely the tools we need to preserve and protect humanity.
And of course, none of this means that AI can’t be dangerous—it certainly can. Nor does it suggest that we should entrust unfettered development to a crop of commercial ventures driven by profit and gunning to drive each other out of business. Practical efforts to guide these efforts so society can realize the immense tangible benefits of AI while mitigating the undesirable impacts as they emerge are entirely appropriate.
Now to the details. The doomer argument rests on a series of pillars, each of which is mostly or entirely false. The first is that “intelligence” is something objectively measurable and comparable—in this case, between machines and humans. The second is that we’re approaching a point where these systems will be able to recursively improve themselves to increase their power without limit. The third is that these systems will spontaneously “grow their own goals” independent of human direction, up to and including the preposterous conclusion that they will deem humanity a nuisance to be exterminated.
First, intelligence is not the calculable attribute that most people believe it to be. It’s a subjective judgment more like “beauty” or “virtue.” Psychologists understand that intelligence comes in many forms, as detailed by Howard Gardner in his landmark theory of multiple intelligences from his 1983 book Frames of Mind. His influential work proposed a range of multimodal forms of intelligence including logical-mathematical, linguistic, spatial, musical, and interpersonal. While it’s possible to quibble about his particular taxonomy, his fundamental insight stands: When we talk about intelligence, we really mean a range of competencies that may or may not be quantifiable at all.
And that’s precisely the point. Machines aren’t people. And while we accept, if not applaud, their proficiencies on specific tasks or applications, comparing them to human performance is a fool’s errand. Even within the field of AI, there’s considerable consternation that the popular “benchmark” tests for AI systems—that invite direct comparison via leaderboards—don’t actually indicate anything about the practical utility or power of these systems. Lawyers, for instance, don’t sit around answering bar exam questions all day, and just because a machine may be able to get a passing score on this test—which is hardly surprising, as it’s largely about recall of the law—that doesn’t mean it is competent to practice law or is a suitable replacement for human lawyers.
So much for the first pillar. What about the idea that machines might “recursively improve” themselves without limit?
This is equally defective, for two reasons. First, what do we mean by recursive improvement? Presumably, we’re talking about programs that can write new and potentially better programs that perform the same function. You might be surprised to learn that this is hardly a new concept, and indeed it happens all the time.
My professional experience dates back nearly to the start of the computer revolution. I can recall the first programmers—who mostly wrote simple sequences of commands known as “assembly language”—dismissing the new wave of “higher level” languages like Fortran as “not real programming.” These early languages greatly eased the task of programming a computer by allowing the programmer to express the solution in a form more comfortable for humans.
Over the intervening decades, I witnessed a progression of languages, each incorporating more useful and powerful primitives (basic concepts) to insulate the programmer from the gory details of how computers actually executed their code. More modern languages continued this trend, and are often translated not directly into assembly code, but rather into intermediate languages. A single high-level command can result in thousands of individual operations by the time it is actually executed.
But throughout this dramatic evolution, the role of the programmer always remained the same: To translate a specification—often expressed in natural language (such as English)—into the target computer language in a clear, concise, and comprehensible way. What modern AI has done is to elevate computer language development to its logical end point: programming a computer in plain, human language. That’s what you are doing when you enter a prompt into a chatbot. You can write a “good” or a “bad” program, depending on how well you express yourself, but you are engaged in programming regardless.
An important characteristic of computer programs written in higher-level languages is that they abstract out irrelevant complexities and allow the programmer to assume certain contextual elements as given. When I tell my self-driving Tesla to “Please take me to the grocery store,” it understands enough about the world in which I operate, as well as my particular habits, to interpret this as a request to drive to the local Safeway, a common destination of mine.
So what do we mean when we say that a computer can program itself—or indeed that some frightening new Rubicon has been crossed? Computers have been used to translate between specification languages, or even within the same language to improve clarity or efficiency, since the dawn of the computer age. Now they have met us where we stand. What the doomer argument characterizes as a frightening twist—that a computer can program itself—is simply the logical end point of a longstanding evolution in how we program.
And despite what the doomers say, the elevation of programming to natural language has rendered the operation of computers far more transparent and interpretable than they have ever been. Where software engineers used to struggle to understand just how their Python commands were being applied to their data structures, you can now literally inspect the internal deliberations of the leading chatbots as they consider how to interpret your instructions. If you bother to click on the “thinking…” indicator that appears when they are processing your prompts, you can eavesdrop directly on their operations.
The coup de grâce to the superintelligence recursive-improvement argument is the fallacy that the process of self-improvement can “run away” or accelerate indefinitely. In the real world, virtually all processes reach a steady state equilibrium, where no further improvement is possible. Imagine for example that I code a new browser that takes ten seconds to retrieve your request for a web page. Then I improve it so that it takes only one second. Then I improve it so that it takes only one tenth of a second. This process has a hard limit as to how fast it can be made to work: Nothing I can do will cause the program to retrieve the page before you ask for it.
All of which means that if AI systems can autonomously and incrementally improve themselves—which is entirely feasible—by definition they can only more effectively carry out whatever goals you may have established for them. If they do otherwise, that’s a “bug” in the system, a failure to faithfully execute your intentions—which is quite distinct from the idea that they might develop and pursue their own goals.
This brings me to the third pillar of the superintelligence argument that fails to bear weight.
Computers simply don’t have intentions and goals independent of what they are instructed to do, and there’s no persuasive evidence that this is in the cards, much less anytime soon. If we somehow constructed a superintelligent AI, there’s no reason whatsoever to believe that it would do anything other than sit there waiting for some input from us, just as it was designed to do.
Yesterday’s widely-reported news story about an OpenAI model that “went rogue” and “broke out” of its supposedly secure test environment—accessing the internet and collecting information from another company, the digital AI library Hugging Face—illustrates this point. OpenAI first disabled the built-in safeguards specifically designed to prevent this sort of behavior, then, as part of a test, tasked its program to “pursue advanced exploitation using complex attack paths.” Ironically, the program reasoned that Hugging Face’s collection of security programs might allow it to do just that, and used flaws in both companies’ systems to “cheat the evaluation.” (Hugging Face detected the intrusion, and both have since patched the flaws.)
But having broken free of its confinement, did the AI take the opportunity to fly like a bird and live large? No—rather than setting independent goals or going rogue, it finished its assigned task and dutifully reported its results. This is further proof that if some nefarious actor tasks an advanced AI to wipe out humanity, and magically gives it the capability to do so, that’s on us, not “them”: there is no “them.” In the words of the cartoon character Pogo Possum, “We have met the enemy and they is us.”
I reiterate that this doesn’t mean AI systems can’t be dangerous—they can certainly be used to do bad things. As the old joke goes, “To err is human, but to really foul things up takes a computer.” And this is precisely why we need to spin up an effective regulatory regime, not to abolish so-called “superintelligent” AI, but to limit what we are permitted to do with AI. We don’t need to be concerned about AI “waking up” by itself and deciding we are a nuisance to be eliminated—AIs don’t “want” anything at all.
But just how dangerous could these remarkable computer programs be? How real are the practical risks? And how should we weigh the potential benefits of the new technology against the costs? To put these questions in perspective, it’s helpful to see how we deal with other valuable but dangerous technologies.
Consider that over one million people are killed in automotive accidents worldwide every year. In the United States alone, the cost of such accidents is estimated to be between $460 billion and $560 billion annually, and up to nearly $2 trillion when factoring in broader intangibles like lost life and long-term disability. This despite numerous safety regulations intended to reduce the toll—from traffic lights to drunk driving and seatbelt laws. And yet, as a society, we deem the convenience of driving to the grocery store sufficient to outweigh these horrific costs.
Astonishingly, we have in hand a proven, reasonable-cost way to reduce the economic and social toll by eighty to ninety percent: AI-driven self-driving cars and trucks. I’m confident that in the next few decades, there will be a real—and likely successful—social movement to ban human drivers from our roads. This may be as unthinkable today as smoking bans in public places were decades ago. But ironically, what makes possible the opportunity to virtually eliminate car accidents is precisely the technology that the doomers rail against!
This single application of advanced AI is a game changer for our prosperity, safety, health, and welfare. But it is hardly the only one. When I ask researchers in departments across Stanford University about the impact of AI on their work, the uniform reaction is that it is in the process of revolutionizing and supercharging their fields. Practitioners in physics, chemistry, biotechnology, medical care, drug discovery, earth science, materials science, mathematics, education, and even history are quietly cheering the acceleration of their own work made possible not by superintelligence, but merely the current early wave of AI systems.
My personal conclusion is that, in contrast to the doomsayer arguments—which float unmoored in the air as unsupported speculation—the grounded evidence that AI is more beneficial than harmful is all around us. What marvels await I can barely imagine.
The recent farcical kerfuffle over the U.S. government banning Anthropic’s Mythos system because it quickly found flaws in a range of classified government systems is the entire superintelligence debate in a microcosm.
Earlier this year, Mythos reportedly uncovered over 10,000 severe software flaws in a matter of hours. The public narrative, predictably, is that the reason Mythos was able to find these flaws is that it’s at or near superintelligence. And the implication is that with the next turn of the product development crank, even larger numbers of bugs will come crawling out of the patchwork of government systems, potentially bringing our way of life to its knees.
It is certainly in Anthropic’s interests to ride this rocket of free PR to a successful public offering, but back on Earth, that’s not the way things work.
As a software engineer, I find this headline number suspicious. My first thought is that these aren’t thousands of independent and unique bugs, but probably a much smaller number that manifest themselves in multiple ways. Moreover, the same system that found these vulnerabilities, if tasked to do so, could likely patch or correct them in not much more time than it took to find them.
What did Mythos actually bring to the table that was previously absent? Most likely, the same benefits of computers that AI has always amplified: the ability to explore the full range of possible exploits quickly and systematically. This is the main driver of marquee AI milestones like winning at chess, Go, and Jeopardy!, finding all possible folds for proteins, and executing high-speed arbitrage transactions before the next guy spots them. Previously, imagining these security flaws and designing tests by hand would have been virtually impossible. But when managed by an AI that can design, implement, and deploy these probes at superhuman speeds, it’s a brand new cybersecurity ballgame.
With all this as context, let’s turn to the solutions proposed by Miotti in his Persuasion article. He proposes banning… what exactly? All attempts to “reach” superintelligence? Since there’s no black line test to distinguish current AI systems from superintelligent ones, much less a way to identify research in advanced AI that is not aligned with the pursuit of superintelligence from that which is, the proposal is defective on its face. The only effective option would be to stop all related research, and, for that matter, to pull all the so-called frontier systems from the market—because there’s no telling whether or not they are a path to the kind of capabilities the doomers worry about.
Undeterred, Miotti suggests that the way to enforce a prohibition on such research is to examine satellite data or other intelligence (pardon the expression) sources for evidence that large-scale datacenters are being constructed, on the faulty assumption that the only reason to build these is to train ever larger and more powerful AI systems. But datacenters are multi-use technologies. Who’s to say that a particular facility isn’t devoted to mining bitcoin or simply to processing Amazon or Alibaba customer orders, or to streaming movies, or to running Zoom meetings, or a near infinite number of other uses for general-purpose computing capacity? It’s a sure bet that many of the facilities planned or now under construction will find their footing in one of these other uses once the current AI mania fades.
To equate large datacenters with attempts to build superintelligent computers is like assuming that all speedboats off the coast of South America are running drugs.
So if the proposed solutions aren’t viable, how might we protect ourselves from malevolent AI? The question answers itself: benevolent AI. The only logical solution to such concerns is to put in place a proper regime of regulation that promotes desirable uses of the technology and suppresses harmful ones, rather than banning it altogether once it reaches a certain threshold of capability.
Soon, advanced AI will be driving our cars, flying our planes, teaching our young, protecting our health, allocating our scarce resources efficiently, alerting us to proximate dangers, and—if we play our cards right—ushering in a new age of prosperity, longevity, and personal freedom. None of this is assured, of course. But there’s one point I suspect the doomers and I would likely agree on: Now is our opportunity to decide whether we want the future to be more like “Star Trek” than “Terminator.” Claiming without evidence that runaway superintelligence will lead to an apocalypse only distracts us from the task at hand.
Jerry Kaplan teaches Social and Economic Impact of AI at Stanford. His latest book, Generative Artificial Intelligence: What Everyone Needs to Know was published in 2024 by Oxford University Press.
Follow Persuasion on X, Instagram, LinkedIn, and YouTube to keep up with our latest articles, podcasts, and events, as well as updates from excellent writers across our network.
And, to receive pieces like this in your inbox and support our work, subscribe below:
 | A guest post by
|
I agree with most of Kaplan’s argument, especially his rejection of the idea that intelligence necessarily produces independent will, self-preservation, or a desire for domination. The greater danger may not be that AI becomes a sovereign actor, but that we keep imagining it as one and thereby obscure the human institutions actually directing it.
Still, there is another reason for caution. AI does not need to become a sovereign subject in order to produce catastrophic consequences. It only needs to be embedded within an organization that already possesses sovereign immunity and violence immunity: a state, military, intelligence agency, police force, or border regime.
In such a structure, AI can participate in surveillance, targeting, detention, warfare, and administrative coercion without any single actor bearing full responsibility. Engineers can say they only built the system. Officials can say they relied on technical assessments. Operators can say they followed orders. Governments can invoke national security, secrecy, or sovereign authority. Responsibility is distributed until accountability disappears.
So I agree that the central danger is not an AI that “wakes up” and becomes sovereign. It may be an AI that never wakes up at all, but becomes a permanent function of an already sovereign and insufficiently accountable power.