Superintelligence Isn’t Enough
AI has many answers—but it can’t by itself build a new society.
Just so you know: In addition to this column, Francis Fukuyama also writes for the main Persuasion mailing list. That means that if you don’t subscribe to Persuasion, you might miss out on receiving some of his essays!
Click here to manage your account and toggle on the button to receive emails for Persuasion. And as ever, thank you for supporting our community.
I have a longtime friend whom I’ve known since my college days, who made his money as an investor and entrepreneur at the edge of the tech world. One constant about him over the years has been his endless admiration for people he regards as “very smart.” He means this in a very specific way: they are very good at math, and have done well for themselves making money using their brainpower.
He’s not alone in this preoccupation. Silicon Valley is a virtual cathedral for the worship of geniuses—initially people like Steve Jobs, Bill Gates, and Elon Musk—who have built world-beating companies around applications of technology. That technology has now moved onto AI, where Sam Altman, Demis Hassabis, and Yann LeCun have become the new icons of brilliance.
And what this generation is building is, indeed, intelligence. There is a race currently on for artificial general intelligence (AGI), a machine that will have the cognitive capabilities of a human being. Indeed, more than that: cutting-edge machines are “growing” rather than being programmed, and are reportedly capable of modifying themselves to extend their own capabilities. They will not stop at human intelligence, but will become smarter than human beings. This type of “superintelligence” will then lead to huge advances in science, technology, and the economy. There are already achievements along these lines, like Hassabis’ Alphafold project that has solved protein-folding problems that seemed beyond the capabilities of earlier technologies. There are serious discussions taking place now about a future, not that far away, where advanced economies using superintelligent AI will be able to achieve substantially higher growth rates of 10, 15, 20 percent per year, compared to the 2-3 percent that’s considered substantial today. Material deprivation will disappear and be replaced by schemes to subsidize those whose livelihoods have been displaced by AGI, like universal basic income.
There are several problems with these speculations. The first is one I’m not in a position to evaluate: whether AGI or superintelligence are even possible. Writers like Eric Larsen have suggested that while LLMs are good at culling enormous stores of existing knowledge, they lack the kind of speculative insight that the cognitive scientist C. S. Peirce labeled “abduction” that is required for true innovative discovery.
But let us assume for the moment that AGI will come about, and that machines will become more intelligent in certain respects than human beings. There are powerful reasons to believe that this capability will be transformative in many ways, but may not produce explosive economic growth as the AI cheerleaders expect.
The reason for this skepticism is that the binding constraint on economic growth today is simply not insufficient intelligence or cognitive ability. Even absent smart machines, human beings today collectively have more cognitive ability than at any prior point in human history. The binding constraint has to do with how that intelligence interacts with the material world in a myriad of ways. Economic growth depends ultimately on the ability to build real objects in the real world. A smart machine may be able to come up with a plan for a better mousetrap, but to actually fabricate that mousetrap requires capabilities beyond any machine’s control.
At a macro level, we are already running into the constraint of too many dollars chasing too little stuff. As environmental doomsayers have been arguing for years, there are ultimately material limits to growth. The one most obviously in front of us is global warming, but there are many others. The planet does not have the resources to sustain 8 billion people with an American standard of living; indeed, at 10 percent annual growth, China, America, and Europe would soon run out of everything, including agricultural land, water, energy, and almost everything else.
At a micro level, there is a problem translating the work of smart machines into material goods. Product innovation has always depended on a prolonged iterative process whereby a designer tries out ideas, fails, and modifies the design in response. No amount of superintelligence will ever be sufficient to simulate the behavior of material objects under the conditions of the existing material world, as generations of builders and tinkerers know.
Finally, there is the political and social level. I attended a presentation by an engineer at a leading AI firm who suggested that in the near future, AGI would be able to, for example, provide clean drinking water to struggling cities in the developing world.
The problem is that the failure to provide such basic services in poor countries is not lack of knowledge of what a good municipal water system looks like. The problem is political and social. People do not want to pay the higher costs engendered by a new water system; unionized workers in the municipal water authority do not want to lose their jobs to automation; business owners do not want the disruption that will occur as the streets are torn up for new pipes; the finance minister believes there are other priorities and can’t raise taxes to pay for a new system. In many poor countries, there are water mafias that buy water where it is cheap, and resell it at extortionate prices. They are armed and ready to use violence if you get in their way.
A superintelligent machine may be able to understand these problems, but it will have no way of overcoming them. We already know what a good municipal water system looks like; what we don’t have is an implementation plan to put it in place in city X.
Our understanding of the role of intelligence has been distorted by the kind of technological change that has occurred over the past couple of decades. The internet, social media, and related technologies are all based on software. Apart from data servers and cloud storage, they don’t require fabricating new devices that have never been tested. As a result, software scales very easily. This is how Google, Meta, and other companies have been able to turn into giants so quickly.
Companies that make money by building material objects in the material world have much more difficulty scaling up. They too benefit from economies of scale, but reach a point of diminishing returns much faster than a software company. (This is, by the way, one reason why Elon Musk’s Tesla has been such a remarkable success story, because it has scaled successfully making material products.) We have somehow come to see the software paradigm as the dominant one that will characterize the AI age, but the economic benefits AI promises will not scale so easily.
This is not to say that AI will not lead to huge productivity gains: take a look at Jerry Kaplan’s predictions about the future of robotaxis. But intelligent people, like those in Silicon Valley, tend to overestimate the importance of intelligence in life more generally. There are many other abilities beyond intelligence that make for a good and successful human being, and many other inputs other than what AI can provide that are required to produce economic growth.
Francis Fukuyama is the Olivier Nomellini Senior Fellow at Stanford University. His latest book is Liberalism and Its Discontents. He is also the author of the “Frankly Fukuyama” column, carried forward from American Purpose, at Persuasion.
Follow Persuasion on Twitter, LinkedIn, and YouTube to keep up with our latest articles, podcasts, and events, as well as updates from excellent writers across our network.
And, to receive pieces like this in your inbox and support our work, subscribe below:
When enough people decide they have had enough, they will rise up and force their governments to act. This is not always a violent revolution; sometimes it is just street protests that force the existing power structure to change. The spread of the internet has fueled discontent because people with limited means can see how others lead better lives. Couple this with the knowledge that their leaders are leading lives of ease, and you will see change. This push for change will come from the younger generations who want better lives for themselves and their families. Change is easier in a democracy, with free elections, but even under dictatorships, change is still possible. If AI generates new ideas and capitalists identify profit opportunities, this will prompt more rapid changes. Even in America, where a chaos-monkey leads the government and denies climate change, green energy projects are on the rise because capitalists see the opportunity to make profits. The need for power is rising, and quickly supplying what is needed can only be achieved with green energy projects. The desire for a better life is a powerful force for change.