Arvind Narayanan is a professor of computer science at Princeton University.
In this week’s conversation, Yascha Mounk and Arvind Narayanan discuss why AI’s transformative impact will unfold over decades rather than months, whether human accountability can survive the rise of AI agents in the workplace, and what the economy will look like once AI has automated every task that can be precisely specified.
This transcript has been condensed and lightly edited for clarity.
Yascha Mounk: You have coined a phrase that really is central to some of the debates about artificial intelligence, but what you mean by it goes a little bit beyond what you would intuit just by looking at the phrase. The phrase is “AI is normal technology.” What does it mean to think about AI as normal technology?
Arvind Narayanan: “Normal technology” doesn’t mean this is mundane, boring, nothing to see here, move along, we’re not AI skeptics. The paper starts out by acknowledging that this is, and is going to be, a transformative technology, perhaps on the scale of electricity, the various industrial revolutions, etc. We do think that it’s a transformation for cognitive work in the same way that in the past we had transformations for mechanical work. But I think that historical analogy gives us a lot.
We do think that there are deep lessons from history to think that just because this technology is so powerful, it’s not going to transform the world over a period of a year or two. There are many technologists, tech leaders, and business leaders predicting that just because numbers are going up on the charts they’re looking at, AI capability charts, it’s going to make human labor unnecessary, put people out of work, even end the concept of money, or pose existential threats to humanity. We think these are all valid things to worry about. We’re glad people are doing research on them. But we think there are so many bottlenecks between AI capability increasing and it having these impacts, both good and bad, especially if you want to really integrate it into the economy and derive something useful out of it. There are so many things that have to go right. We think those are going to happen over a period of decades, not months or years.
We have a lot of collective agency in shaping how AI is going to transform various professions, the economy, democracy, etcetera.
Mounk: I find this to be a really useful way to think about this. Again, you’re very careful to say you’re not an AI skeptic. I guess I’m a little unsure about what exactly we mean by the term “normal” in this context. When you look at electricity, even when you look at the internet, these were technologies that did transform the world in a very fundamental way.
Now, when you’re comparing that against the most naive kind of pronouncements that some people in Silicon Valley are making, where in eighteen months, everything is going to have changed—I remember sitting in a conference room with somebody who’s now very famous in Silicon Valley in 2017, pointing out of a window at a green field and saying, in eighteen months, robots are going to be building skyscrapers over there. It’s 2026, and the housing crisis in California is as bad as it ever was. So I’m very sympathetic to that point. But even if this is going to take decades rather than years, and the way in which the technology shapes the world is obviously going to be inflected by all kinds of human choices, institutions, and regulatory obstacles, something on the scale of electricity still feels, in some senses, pretty abnormal.
Narayanan: It is a technology to be taken seriously. I think individuals, businesses, and policymakers should all be taking it seriously, and that’s why we have been working in AI. My co-author Sayash Kapoor and I have a long history of advising policymakers on how to respond to these changes. If we weren’t taking it seriously, we wouldn’t be spending our own careers doing all this work. But to put my finger on some of the disagreements—exactly like you pointed out—there are so many claims that this transformation is going to happen over a year or two. And if that’s the case, then all the kind of more normal policy levers that we might have—help people find new skills given this new technology so they can find new jobs—all of that goes out the window because nothing can happen fast enough on this time scale. The only thing we can prepare for is mass joblessness. So we should be talking about anything less than the scale of universal basic income being completely disproportionate to the scale of the problem. That’s exactly what tech leaders are advocating, and we’re strongly against that.
We do think the normal tools of policymaking, business, and human adaptation can work with AI as well. We do need to fix some problems. Even “normal” policymaking, even when you have a technology that’s not advancing as fast as AI, often takes a very long time to acknowledge problems. With social media, for instance, it took well over a decade after it started having these massive impacts on society for the research to actually get there, for policymakers to wake up, etcetera. Those are problems we’ve had, and we’re going to have those problems with AI as well. We do need to improve the speed at which we respond to these changes, but it’s not as if it’s a one or two year thing and we have to throw out our entire existing playbook and prepare for either some utopia or catastrophe.
Mounk: Let’s distinguish between the speed with which the transformation is going to happen and the scale of the transformation that will ultimately come. I think we strongly agree about the speed. Having read a lot of your work, I’m not yet sure whether we agree about the scale. But let’s start where it’s easy, which is the speed. Why, in precise terms, do you think the impact of this technology is going to take much longer to materialize in the real world than a lot of these people in Silicon Valley seem to believe? Why is this a question of decades rather than years or even months?
Narayanan: Let’s take a concrete example: software engineering. That’s the area where AI adoption has been most rapid, and AI capabilities are furthest along. There has been this prediction that AI is going to automate software engineering once AI can write all the code. We already have enough evidence now—we just put out an essay on this—to reject this model. AI is writing most of the code now already, and it has not made software engineers obsolete.
Software engineer hiring—the number of employed software engineers is still growing, maybe slightly slower than before, but even that is disputable, because it’s also allowing software engineers to more easily become entrepreneurs, for instance, and that’s not quite captured by the data. We looked into why this is, and we have a model to explain it. It’s what we call the “decide, execute, deliver sandwich.” These are three layers that we think apply to most kinds of cognitive work.
Cognitive work—thinking for a living—is the kind of work people are saying is going to get automated with AI. Our analysis is that most of those jobs, whether you’re a software engineer or a lawyer or a researcher, or many other kinds of cognitive work, a good third of it is just figuring out what the problem is and making decisions about whether the problem is worth solving, how we’re going to solve it, how we’re going to build the software, design the system, etcetera.
Then maybe another third of it, again in the software engineering context, is writing the code, fixing the code, etcetera. And another third of it is delivering the code—verifying it, making sure it passes quality checks, being accountable for it, integrating it into the customer’s systems, maintaining it over time, especially when you’re thinking about enterprise software rather than consumer software. This is a big chunk of it. What we’re finding is that AI is compressing the middle of the sandwich. It can do the writing of the code, but the other parts of it that require a human to be accountable for what is ultimately delivered—it has not taken the human out of that process. If anything, those layers of the sandwich have expanded to fill the gap of the time savings that you get from having AI write all the code. There’s more to say, but I’ll stop here, and I’m curious for your thoughts.
Mounk: To what extent is that a function of the stage of the technology? ChatGPT 3.5 was rolled out, what, three and a half years ago? The progress in what these systems are able to do has been astonishingly rapid. Clearly, on the core task of writing code, they now rival the very best human engineers. But of course, in terms of putting in place the structure for AI agents to be able to reflect on what actually is a worthwhile problem to solve within an organization, or thinking about how you set up AI agents that are actually able to reliably test the quality of code in the context of the company’s needs, we’re not there yet. That certainly speaks to your point. But what is to say that they won’t be rolled out over the next three, five, or ten years?
More broadly, when you’re thinking about a lot of these essentially political demands within a corporate context—somebody needs to be responsible. If this leads to some terrible outage, you’ve got to have somebody you can fire, somebody you can blame. That’s certainly true in current organizations, and it’s very hard for those organizations to reform themselves. But surely, when we think about a longer timescale, we might imagine AI-native corporations that have a totally different setup, one much better able to use AI technology to replace some of that human judgment.
Narayanan: That’s a great question, and I think this is a key point of disagreement—this is where we disagree with a lot of our critics who might agree about the timescale but disagree about the scale of transformation. Two points. Like you said, political demands—I don’t think these will change simply because the capabilities improve. One of our key points is that we have agency to insist that humans ultimately remain accountable for what is delivered. That might be true even if, in some narrow sense of capability, AI could do a better job, precisely because AI is not something you can punish, so to speak, or keep accountable. It does not have some of the limitations that humans do. I think the right thing to do is to insist that humans remain accountable. There is something inherent about human accountability that you cannot get with AI. That’s not up to the AI companies—that’s up to the downstream organizations who are deploying AI, and, if it becomes necessary, regulation to insist that this be the case. This is what we’ve done historically with many technologies that are very powerful but dangerous. I like the analogy of a crane operator. The crane can do the heavy lifting in construction—we don’t need human physical labor. But even though we could, we don’t let the crane operate autonomously. We put an operator in it, and the crane becomes a massive amplifier of human ability. That’s the model we think is the right one, and we can choose to keep it that way. We can’t guarantee that it will be that way, but I don’t think it’s a matter of the tech companies themselves deciding.
We hope you’re enjoying the podcast! If you’re a paying subscriber, you can set up the premium feed on your favorite podcast app at writing.yaschamounk.com/listen. This will give you ad-free access to the full conversation, plus all full episodes and bonus episodes we have in the works! If you aren’t, you can set up the free, limited version of the feed—or, better still, support the podcast by becoming a subscriber today!
If you have any questions or issues setting up the full podcast feed on a third-party app, please email leonora.barclay@persuasion.community
The second point is where a lot of these disagreements come from. We do think it’s true that gradually, over time, the set of tasks you can delegate to AI is going to expand. But what that means is that once AI can do something, it’s no longer a source of competitive advantage in the enterprise. It’s not a fixed amount of things we’re trying to build, and then once you can build those things with AI, you’re done, there’s no need for work. That’s not how the economy works. Once you can automate certain things, delegate those things to AI, that becomes a common capability that every firm has. Then what firms are competing on is what is scarce. Human labor will always be scarce, because you can’t make infinite copies of it. So those remaining pockets of judgment that AI is not yet able to perform, that is what will be in the domain of human judgment and expertise, and that’s what companies will compete on.
In the distant future this might change, several decades down the line, but for a long time we’re going to be in this period of a constantly upward-adjusting equilibrium.
Mounk: I take these arguments very seriously. Just two observations. The first is that you slipped in “several decades.” I’m trying to think about this at the timescale of history and at the timescale of the challenges our society is going to face. Are people who are today in their 50s or 60s likely going to be fine? I think so. Are people in their 30s and 40s going to be fine? I think it’s a more open question. What is the world going to look like for people in their 20s today who are about to go to college? I think that is much, much more challenging. Insofar as the answer is it’ll take several decades, that is reassuring in some sense, and it certainly is totally in line with the evidence from the Industrial Revolution, from how long it took for the invention of the printing press to have a major impact on European culture or society. I absolutely buy that. It’s just not clear to me that several decades is as reassuring as it might be. Go ahead.
Narayanan: Sure. Look, the “AI as normal technology” framework is reassuring to some people, and it’s not reassuring to others, and our goal is to give this framework for people to do with it as they will, not necessarily to provide reassurance. The reason it’s reassuring to me is because over a several-decade timescale, I do think people can adapt. Almost everyone can adapt. If it’s one or two years, then most people can’t adapt. If it’s several decades, then we can use game theory, if you will, to think about what that equilibrium is going to look like whenever it happens, and try to see what we can start doing now to prepare for that.
One way things might turn out—and the equilibrium part is something we’re only now starting to think about, it’s not in the essay itself—is that any task in the economy that you can specify precisely enough for it to even be legible as a task is going to be done by AI. So most of the things now that make up my job—okay, my job consists of these ten things—all those ten things are going to be done by AI. What jobs will mean in the future is what economists sometimes call interstitial tasks: the things that lack a precise specification, where maybe part of the task is even to figure out what should be done next, things that maybe have never been done before. Those are the kinds of things we think will be in the domain of humans. The reason I think we will be very unlikely to leave those things to AI is because of unknown unknowns. You can’t be sure that when you task AI with doing something so different from anything that’s been done before, it’s going to do a halfway reasonable job. That’s one way to think about what a new equilibrium might look like.
Another, perhaps even more radical shift, is that maybe everything that’s a commodity—and a commodity, not just a physical good, but even cognitive labor—a commodity is one where you don’t care which particular human or AI produces it. A lot of what we produce now are commodities. I produce research, I write these articles—it doesn’t matter that it came from me, it’s really mostly the ideas that matter. Maybe that’s mostly going to be done by AI in the future. We should be prepared for radical shifts, like professors having a lot of status because we can produce something that is unique today might not be unique in the future. There’s going to be no need for people doing podcasts like this—this sort of stuff is all things you can get from AI, maybe. I think we should be ready for that.
Mounk: I think the three Ps are always going to survive: professors, podcasters, and prostitutes. But we’ll see.
Narayanan: Nice. I’ll bet against that. But those things are very different. I think some of those services are very different from others. Sex work, for instance, is inherently relational—it’s about a particular person. There are many kinds of jobs in the economy. Even some low-status jobs, like barista, for instance—Alex Imas has nicely written about why some of those jobs have been very resistant to automation, because human interaction actually matters there. What the economy in the future will look like, maybe, is basically people paying each other for their time, for the unique relational value they can give. The value of going to a coffee shop will be the customer service. If you have good coffee and shitty customer service, that’s totally useless, because robots can make good coffee, but it’s the good customer service they can’t replace.
Mounk: Let’s take a step back here, because there are a lot of really interesting ideas in this. A few thoughts. The first is, it’s not clear to me why an AI system that at this point is very good at writing short stories—we’ve recently learned that an AI-generated text won a major literary prize, with judges being unaware of this fact at the time—that increasingly does have good judgment about medical conditions, for example, looking at a lot of contradictory data and coming up with the best kind of adaptation of what’s happening—won’t also be able to learn these kinds of interstitial tasks. So there is some kind of future where AI systems actually prove to be as capable as the most intelligent humans, even at a task that looks slightly ragged or irregular.
The second kind of possibility, surely, is that there are some human beings who have exceptional skills that will still be required. There may, in fact, be some human beings who are able to command tremendous wages because they are so good at orchestrating these AI swarms to accomplish tasks relative to others. But it may be that the number of people capable of those kinds of cognitive skills remains very limited, leading to a kind of class or caste of super-earners from AI, and a large portion of the population whose skills are much less valuable.
A third kind of possibility—and to some extent you can mix and match these—is to think that perhaps there’s still a need for relatively ordinary human skill. Obviously, being a great barista or a good bartender is a genuine skill, and I admire the people who do it, but it’s a less rare skill than somebody being able to code at the very highest levels was a few years ago. Perhaps the number of positions we need for that is just much less than the extent of available human labor. It may be that nine out of ten times, I’m very happy to take my excellent robot-made or coffee-machine-made coffee for 10 cents, and once a week I want to go and meet somebody for coffee in a really pleasant environment, and I’ll definitely want a super cheery, nice barista to make my coffee on those occasions. But that just may not create enough work for people to keep going. And in that kind of world, there’ll be an army of people who are out of work who are very happy to take that job as a barista, depending on the kind of socioeconomic arrangements and background conditions that hopefully would cushion poverty. In that case, even the people whose skills are still in demand may not make a very good salary.
I think here we need to be careful not to go into a world of either no human being ever is going to have a job anymore—which I think the two of us are going to agree is a kind of weird Silicon Valley dystopia, though some would probably call it a utopia—but then to say, well, if that’s a one, then the zero is some jobs are still going to be needed, and so therefore the problem is solvable. There are all kinds of worlds in which certain kinds of jobs are needed, but they’re not the ones that create the most economic value, they’re not the skills that are actually scarce, and therefore they fall very far short of creating the structural preconditions for an affluent middle-class society.
Narayanan: Yeah, look, all of those are possible. And I’m glad that we’re talking about them. People are looking at leading indicators for which way things are trending. I do think these are solvable problems over a multi-decade time frame. Over that kind of time frame, even if the answer turns out to be universal basic income, we can plan for that over the long term. That’s not something we can make happen in 18 months, but it is something we can make happen over a several-decade period.
I genuinely don’t think something that radical will be required. I agree with most of what you said. Maybe the key disagreement is that I don’t think the set of jobs that are necessary in the economy versus not necessary is that helpful, at least to me. In my view, the vast majority of jobs today are already doing things that are not near the bottom of Maslow’s hierarchy. The things that we need, like agriculture and shelter, that’s maybe something like 5% of jobs. And all of the other stuff—if you just took yourself away from the everyday intensity of life and the economy and really thought about it from an alien’s perspective—it’s all just stuff we do to amuse ourselves. I really like David Graeber’s book on this.
Mounk: I do have to say, sometimes I walk through the streets of Brooklyn and I say, there’s so much wealth here. What does everybody do? What makes this perpetual motion work? It’s just people selling each other goofy services and somehow living well off of it.
Narayanan: In many parts of the world there’s obviously genuine crushing poverty. In affluent societies, 95% of jobs are things we do because we need things for people to do, not things we do for people to survive. I’m saying that in the future, that 95% will go to 99%, maybe even 100%.
Mounk: I’m torn on this. I do have that feeling walking through the streets of Brooklyn, but when I think about this example a little more carefully, one of the huge reasons why New York is so affluent is that finance is headquartered in New York, and of course many other industries, from fashion to media, have a huge presence in New York as well.
We can have a big debate about whether finance really is a productive enterprise or whether it’s not, but the point is that that is where a lot of the value in society is created at the moment. And in order to run Goldman and JP Morgan and all of those companies, the owners of that capital currently require a lot of scarce and very high-skill labor. One of the reasons the streets of Brooklyn are full of expensive yoga studios is that a lot of the people who go to those yoga studios work for those banks, or they work for people who work for those banks. And so this is where the tremendous wealth of the city diffuses.
In an economy in which AI systems produce the huge majority of value, and human beings are then left in the role of educating each other, looking after each other when we’re old, and “putting on concerts” for each other—all of that is kind of peripheral to the core activity. Can that sustain itself? I think perhaps, in a somewhat simplistic way, that depends on whether the first image I painted of Brooklyn is right, or the second image is right. And much as I like to hate on Brooklyn—which I love to be in—I think perhaps the second image is a little closer to the truth than the first, which is somewhat troubling in its implications for what that world would actually look like.
Narayanan: Again, it’s a possibility. I think the reason I’m betting against that is when you look at what actually has value, it’s just very contingent on what happens to be scarce at any particular point in history. Things like light and clothing—things at the bottom of Maslow’s hierarchy—used to be extremely scarce because they were hard to produce, and they employed a very large fraction of the workforce.
Their costs have fallen by a factor of well over a thousand over a period of a couple of centuries, and that’s because of automation, and because of larger-scale energy production and cheaper costs of energy. Well over a thousandfold decrease in the cost of lighting—it could be closer to a million, I forget the exact numbers, but it’s many, many orders of magnitude. I think a lot of the things people do today are scarce and valuable precisely because human labor is scarce. I disagree with the premise that in the future AI systems will be creating most of the value. That’s not consistent with my understanding of economics. I agree that AI systems will be doing most of the work. But any work AI can do is infinitely reproducible, and when it’s infinitely reproducible, it doesn’t have a lot of value, because its cost is going to come down to the market—anybody can provide it at very low cost. So the value is going to shift to the kinds of things that are scarce.
Today, the labor of caring for the elderly, for instance, is very low status and very low paid. But the argument is that in the future, a lot of the things that are higher status and higher paid today—maybe like the things we’re doing—are going to become very easily reproducible. So what is considered valuable in the economy is going to shift.
Mounk: How does that work mechanically? You’re saying taking care of the elderly is going to be the thing that is really valued, and presumably that’ll command a salary more generous than what a professor at Princeton gets today, hopefully. But where does the elderly person get the money to pay the person taking care of them such a handsome salary?
Presumably, the answer is that they themselves must have done something in the economy that accumulated that money for them, or that we have something like a system of universal basic income—which I take it you’re mostly opposed to, and that’ll be interesting to go into—that provides them with the money from a third party. Again, looking at Brooklyn today, the intuition is that it’s a kind of weird perpetual motion machine where everybody’s just selling stuff to each other. But when you look at it more closely, you can say, no, there’s actually a way in which the banks are financing manufacturing companies and agricultural companies and all kinds of other things that produce things, and there are a lot of very well-paid employees at those places, and they’re the ones that explain why people can pay $10,000 to go see the Knicks in the NBA finals.
You take that link to the underlying economy away, because so few humans are required to run that underlying economy—because whatever the most productive firms of the future are, they just require so many fewer human beings working in them. And it’s unclear how that money is going to diffuse into the economy, such that even if socially we say, what a wonderful task, to look after elderly people, that elderly person actually has the money to pay a good salary to the person taking care of them.
A very different way of putting this point: you compare Walmart, which has a valuation of about a trillion dollars—a little less, I think, depending on exactly when you look—and has two million employees, to Anthropic, which probably is going to be valued at about a trillion dollars when it IPOs, and at present has about 5,000 employees. That is a really striking difference, and that surely is going to have a huge impact on who can actually pay for these kinds of human-centered skills to be valued in the labor market.
Narayanan: Yes, it won’t automatically happen. I’m just saying that we will have the means to solve it, even though it will require a lot of work to solve it. The reason I’m saying that is that we’re making kind of fantastical assumptions in one dimension—that AI is going to be able to automate nearly all work—and we’re assuming that the world is going to stay the same in all the other dimensions. I think that’s a bit of a pitfall.
This future might not happen. It might be that even though I’m on the more “slower transformation” side, maybe even I’m being way too optimistic, and maybe there are really fundamental barriers to AI being able to take over a lot of this work even over a multi-decade timescale. But let’s assume that all of that happens. Then clearly it’s going to be creating an incredible amount of benefit to humanity, because so many of our material needs are going to be taken care of. And there is going to be, I think you’d agree, some accumulation of wealth. That’s the distributional question that is unclear. So if greater and greater amounts of wealth are going to be accumulated, we can start putting policies in place. I’m not opposed to UBI. Maybe UBI is what we should be talking about again, over a multi-decade timescale. I just don’t think it’s the sort of thing that can happen if we’re talking about a one- to two-year timeframe.
On the point about Anthropic being a small company that might have a trillion dollars in value—yeah, maybe we’re headed to a dystopian future where only three companies are going to be able to do that. But actually, maybe what’s going to happen is that entrepreneurship is going to become so much easier that there are going to be many millions of millionaires—maybe not Anthropic-level wealthy, but far more millionaires than there are today—because you can have one or two people owning firms that use mostly AI to accomplish a lot of valuable things. And then it’s still challenging, but at least it’s possible to think about, politically and economically, how you distribute that wealth throughout the rest of society.
Mounk: This is a side note, but there is something strange in these discussions where we want to assign this deep value to the kind of human tasks that in one future may still be required. Of course, I do think that relating to humans is something incredibly meaningful, and that often people who are in jobs where they relate to others actually have high job satisfaction. But there is at least something melancholy about imagining a future in which all of the kinds of tasks by which humanity has historically distinguished itself from other animals—tasks we thought were distinctive to us, our ability for higher-order reasoning—go out the window.
Even just in a social sense, the way in which, for the last century or two, for a huge number of human beings, their pride and their sense of self-development came from acquiring an education, acquiring skills, becoming expert at some kind of mental work—and we’re saying the most primal human skills and abilities, to take care of babies and to take care of old people, are now the thing that’s left, and that’s what we really value. I do not mean by this to devalue the incredible work done by nurses, or people who work in old people’s homes, or mothers and fathers taking care of little kids. But it is perhaps less than a wholly full-throated endorsement of a future to say that’s the one kind of thing that’s going to be left for humans.
Narayanan: Yeah, that’s totally fair. Again, I’m not saying that this framework for thinking about the future should necessarily be reassuring to everybody. I think there’s lots that we will lose, that we like about our current society. There’s lots that we will mourn, and I think that’s a very valid reaction to all of this. But again, it is parallel to what has happened in the past with the valorization of physical strength.
If you think about a Roman general, they moved up the ranks economically and politically because they started from their physical strength being very valuable in the army, and that’s how they rose up the ranks. We’re talking about professors in academia—the word “academy” comes from the ancient Greeks.
If I recall correctly, we’ve been talking about professors in academia, and you probably know this better than I do, but the original Academy took its name from a garden where the educational component happened in parallel with the development of physical strength. Those two things were really valued in very much the same way. But over time, we’ve lost that. We still have athletes, for instance, having high-status positions in society, but not because they directly contribute to the economy using their physical strength. It’s just something we like to appreciate in a competition setting, precisely because it’s lost its direct economic value. Now it’s become sports.
Maybe what will happen with intellectual ability, just as has happened with chess, for instance, is that it becomes a sport we enjoy. Similarly, people will in the future present their manually handcrafted ideas without AI in a format we can all enjoy, to appreciate the strength and limits of human cognitive ability. But it’ll be kind of disconnected from the economy, and there’s genuinely a reason to be sad about that.
Mounk: I wonder to what extent the Roman general was prized for physical prowess or for mental skills. In a very different context, about twenty years ago I was dragged to a bullfight, and I was quite reluctant to go, because I thought it would be quite ugly. In part because I thought it would be a kind of macho spectacle about how strong the bullfighter is and how big his muscles are.
It turned out that, at its best, and to my surprise, I rather enjoyed the spectacle. What I enjoyed about it specifically was that I came to understand it was actually a medieval, perhaps only modern, form of celebration of mind over matter—which is to say that the bull is fifty times stronger than the strongest man in the world. There’s no way any human being, no matter how often they go to the gym or how many protein shakes they take, can take on a bull in a competition of strength. What a good bullfighter does is demonstrate that they can use their human skills—their ability to understand how the bull operates, which is that he goes after the red thing—the ability to wave the flag in such a way as to direct the bull’s movement, to do a kind of dance with the bull in which they demonstrate: look what I have is intelligence, and using that intelligence I can prevail against this beast that is vastly stronger than me. The cruelty of it is nevertheless something that troubled me. But I found that to be actually very interesting.
You are right that we have historically valued physical strength a lot more than we do now. But that argument, of course, cuts both ways, because it is now quite hard to make your living from just having physical strength, and the people who do command much lower salaries. When we look at past transformations, like the transformation after the Industrial Revolution, a lot of people were screwed over by them, and this happened over decades and over centuries.
But the thing they had—the thing they resorted to—is what I sometimes call a sort of mental reservoir. There’s a reservoir of demand for human skill, which is to carry out mental jobs, mental tasks, cognitive tasks. So if you were 45 years old and you lost your job in a factory, or weaving cotton, or cultivating a field, your life might be screwed—it might be too late for you to have a good income. But your kids were probably fine, because they acquired those cognitive skills, and that’s what kept them in demand and ultimately gave them a much better standard of living than you likely had. But if we’re now tapping the mental reservoir, is it clear that there is some higher-order skill humans can go to in order to compensate for that? Saying, well, we’ll just go back to the things we’ve always done, like taking care of kids and old people—it’s unclear to me if that’s going to cut it.
Narayanan: Again, I think you’re right to worry. I think there’s a lot that we will lose. I guess if there’s one point of disagreement, it’s that I don’t think that in the future we will distinguish ourselves by our skill. We will distinguish ourselves just definitionally by the fact that human time is scarce, and AI’s biggest weakness is that anything one AI can do is something a billion copies of AI can do.
Scarcity always commands value. That’s how it’s always been. We can create value out of even completely artificial types of scarcity, like wedding rings and diamonds, even when there’s actually no intrinsic value. In one sense that’s very depressing, but again, I think from an economic perspective, we’ll be okay. There’s going to be a lot of demand for human time, for us to have various forms of companionship. We’re going to call it various things—coaching, therapy, care. A lot of professors’ jobs, I think, in the future will just be about managing students’ emotional journey as they undertake their path through the education system. The content, we don’t have to provide—that’s going to become very commoditized.
Mounk: So a good thing that there’s already more administrators than professors at most universities, because really what we need is not professors, it’s these administrators.
Narayanan: Maybe. Look, if this all sounds incredibly depressing, I’m not here to say we should embrace this future. I’m saying a few things. It’s not a catastrophe if everyone being out of a job will have jobs simply because human time is scarce, and we’ll just find artificial ways to turn that scarcity of human time into jobs and call it various things, whether it’s professor or therapist or caregiver.
Lots of wealth will be created. Maybe we’re going to have a problem with a few companies trying to capture most of that wealth, but that is something we can and should change with policy. All of this is going to unfold over a period of decades, not one or two years. So we have time to prepare, both in terms of new things we’ll need to do, but also just getting mentally ready for this future, whether you find it empowering or depressing or anything in between. Those are my core points.
Mounk: I’m pushing back on them, but I think they’re very interesting points. Clearly, one of the things you’ve implied a few times in the conversation, and that you’ve written about more explicitly, is that there are real dangers to thinking of artificial intelligence as an abnormal technology. If we assume everything is going to change in the next two years, and we assume that we have to throw out our usual toolbox of policymaking, you think that has really bad consequences for our ability to govern this technology and for the world. Why? How would we be pushed in the wrong direction if we don’t recognize that AI is actually a normal technology, in the sense you outline?
Narayanan: It’s very simple. There’s this narrative coming out of the AI safety community—and they’re doing great research. I respect all the work they’re doing, where I disagree is on the policy implications and this overarching narrative. That narrative is that AI is getting more and more capable, and it’s going to increasingly be able to do things like hacking into critical infrastructure or creating biological weapons.
So if this technology diffuses throughout society, it’s extremely dangerous in two ways. One, AI systems themselves might turn against their creators, or bad actors might get hold of them and do dangerous things. The way to prevent this is to stop really powerful AI systems from being released to the public without guardrails. There are, I think, a lot of problems with this.
The first is the idea that increasing AI capabilities are automatically dangerous. That’s not what we’ve seen historically. When it comes to cybersecurity, for instance, superhuman ability to find and exploit vulnerabilities in software started a couple of decades ago, and it’s been going up since then, even before modern AI systems. Actually, that’s made software more safe, not less safe. The reason is that defenders, the companies that create software, can use those same capabilities to fix those bugs before they even put the software out there. So generally speaking, these powerful technologies have actually helped us improve resilience rather than worsen it. That should be our default presumption. Maybe there are exceptions in some cases, but we should insist on evidence that increasing capability is actually making the world more dangerous.
That’s the first big area of disagreement: one, capabilities increasing doesn’t mean the world is getting more dangerous. Two, in order to prevent AI capabilities from diffusing widely, you basically have to have authoritarian governments. And three, even if you manage to do that, that’s going to be a temporary relief. Eventually the dam is going to break, and what you’ve done by trying to control AI has not built up the kind of immune system you need in order to live in a world with these powerful systems.
Mounk: Presumably, one of the things you might conclude on the most dire version of this claim you’re arguing against is: we’re about to have these incredible cyber warfare capabilities, rogue actors are going to be able to break into your bank account, and probably break into the database of the U.S. military, and possibly commandeer nuclear weapons. And if that happens, then we need to sideline any kind of historical protections we’ve had for private businesses, but also for individuals, in order to make absolutely sure that doesn’t happen. Perhaps we need to empower our governments with all kinds of capabilities they don’t currently have in order to avoid that kind of worst-case scenario. Is that one of the kinds of fears you have?
Narayanan: Yeah, that’s one of the claims that’s often being made, and we have a different position. I do think the government has to be nimble, adaptable. Some of the things that have happened in the U.S. policy context recently, like having voluntary agreements to have governments test the capabilities of new technology and maybe hold off for a month or so before releasing them, those are pretty low on the ladder of extraordinary interventions. If we do them carefully, I think that can be okay.
But where the problem comes is saying that the way to prevent rogue actors from getting access to this technology is to control it so tightly that we make sure they never get their hands on it. Our view is that the only way you’re going to do this is by having authoritarian governments, and not just one authoritarian government—a kind of world authoritarian government, if you will—because it’s getting so much cheaper each year to build these incredibly powerful capabilities. When you look at those exponential charts, they’re amazing. It’s just Moore’s Law all over again. At most, you’re buying a couple of years, and eventually this technology is going to diffuse into the hands of everybody.
Mounk: One side point here, by the way, which I’m sure you know more about technologically than I do, so I’d love to hear your thoughts on it: our frame of reference is often nuclear weapons. That is an incredibly dangerous, powerful technology that can destroy the world, and we’ve been able, to some extent, to prevent its diffusion through international treaties and regimes of inspection and control under certain circumstances, etcetera. My understanding is that there are principled differences between nuclear technology and the capability to build cutting-edge AI models.
One of them is that nuclear technology requires much bigger physical machinery. The United States is quite good at figuring out, through satellite images and other things, whether the centrifuges in Isfahan, or wherever they are in Iran, are spinning or not. It would be much harder to do that for data centers.
The other point is that there’s always some dual-use issue in nuclear power in terms of civilian use, but broadly speaking, you know what it’s being used for. So much of our civilization already depends on chips and data centers. Literally, some of the most powerful chips used to develop artificial intelligence were originally designed to power graphics for video games. Then actually controlling whether, say, the United States and China had a big treaty agreeing not to develop cutting-edge AI models—controlling, from the American point of view, that China is sticking to this, or from the Chinese point of view that America is sticking to this—would be incredibly hard because of this challenge of dual use.
Narayanan: You’ve exactly captured it. There’s dual use, there’s a powerful economic logic of diffusion of AI capabilities and chips, there’s the observability, and there’s the physical bottleneck. Let me add one more point: there’s no particular threshold at which AI becomes dangerous. Every level of AI capability we’ve had has always brought some dangers with it. You can go back to very recent history, and there are striking moments of panic around GPT-2, for instance. OpenAI didn’t initially release the model because they thought it was so dangerous. It’s a model that today my grad students build over a weekend on a single GPU in order to teach themselves how to build these AI systems. That’s how rapidly the technology has diffused, and we’re so much more powerful than GPT-2 now with our recent models. If you stood up today and said GPT-2 is too dangerous, that would sound so ludicrous. My prediction is that whatever we’re saying with Mythos or whatever is going to look exactly like that a decade from now.
The last point I’ll make is that the good thing about the fact that we have been releasing these systems is that it’s also forced us to improve our defenses, because if some of these models can be used for hacking, it’s forced companies to work with various kinds of software makers to give them better access to these models, so that they can find and fix these bugs before the software even ships out there. If we instead follow a policy approach of nobody having access to the technology in the first place for these dual-use purposes, then we will lose the opportunity to build that immune system, and the dam will break eventually. When the dam breaks, it’s going to enter a world that doesn’t have those defenses in place.
Mounk: I’m very worried about giving the government powers over individuals, and to some extent over corporations, as well. I’m a philosophical liberal, I give great importance to individual freedom, and I think we’re going to face certain kinds of dilemmas in this context that are quite dissimilar to those posed by previous technologies. Going back to nuclear weapons, for example, we obviously need to govern how states use nuclear weapons. One of the scary things is that, from Kim Jong-un—not to compare them entirely—to Donald Trump, some rather irresponsible people currently have their finger on the button, and that’s deeply discomforting.
One thing about nuclear weapons, though, is that while they can destroy and blow up the world, they aren’t very useful for domestic political control. You can’t really threaten to win the next election by nuking some city you don’t like. The scary thing about some of the most cutting-edge AI models, from a political point of view, is that they can be used for domestic forms of repression. That means that if you want to avoid the government having all of its powers, you might say it’s better for OpenAI and Anthropic and so on to keep control over those cutting-edge models. But of course, if those companies have control of those cutting-edge models, and they can potentially break into nuclear weapons, or they can potentially have military applications that are absolutely central to the government’s ability to keep its citizens safe, then we can’t really do that either. So how are you thinking about how to solve these real trade-offs, and perhaps these genuine dilemmas, without allowing the government to have powers that could very quickly become tyrannical.
Narayanan: Simply that it’s not all or nothing. I think we can give the government limited power. So, for instance, pre-release access to these models—the government has access before the general public, in order to, again, find and fix vulnerabilities in critical systems. But that’s very different from giving the government the power to try to carry out a mass persuasion campaign. We’ve written about this, and in our view, the critical bottleneck to the government being able to do that is not the capability to generate text or images or whatever, but rather the distribution channels. In other words, if the government were to control, for instance, social media, that’s a much more dangerous thing than governments controlling AI systems when it comes to the ability for persuasion and propaganda.
When you go back to media theory, naively, back in the ’50s, people used to have this “hypodermic syringe” model—the idea that there can be messages coming from companies or governments, or any other adversarial actor, that are so powerful that if people come into contact with those messages, even subliminally, they will change their view. Now we know that’s not true. Yet in the AI conversation, a lot of the discourse proceeds as if that is the case. That’s not how people get persuaded. People get persuaded because they’re embedded in a social community where those beliefs take root for various kinds of tribal reasons. When we look at how AI is being used for disinformation, for instance, it’s being used by those in power, including politicians, to persuade or trick their own supporters, as opposed to their adversaries, because it’s in that context that it becomes very powerful.
So, really, the problem of controlling AI disinformation and propaganda reduces to the existing problems we’ve had with the quality of our epistemic channels and discourse—what do we do about the level of polarization and tribalism, and those types of things. Those are hard problems. They’re not easy. On the margins, AI is going to amplify them. I don’t think AI is creating those problems, though.
In the rest of this conversation, Arvind and Yascha discuss how artificial intelligence is changing warfare, and how governments, companies, and individuals should prepare for AI’s impact. This part of the conversation is reserved for paying subscribers…
