William MacAskill is a moral philosopher and cofounder of the effective altruism movement. He’s the author of Doing Good Better and What We Owe the Future, and a Senior Research Fellow at Forethought Research.
In this week’s conversation, Yascha Mounk and William MacAskill discuss the arguments for effective altruism, the motivations of Sam Bankman-Fried, and whether AI is a risk to humanity.
This transcript has been condensed and lightly edited for clarity.
Yascha Mounk: You are one of the real pioneers of a school of thought known as effective altruism. I think effective altruism is simultaneously a trivially and straightforwardly true idea and an ambitious and controversial idea. At the heart of it is the idea that it’s good to do altruistic things. When you do altruistic things, you want to think about doing that in an effective way, in a way that actually does a lot of good rather than, as some forms of charitable giving activity, relatively little good. What is at the core of this idea, and why is it an idea that is in some quarters controversial?
William MacAskill: Effective altruism is about taking seriously the many big problems in the world, all the big problems we face, and asking, in light of them, how I can do the most good as an individual with my time or with my money, then trying to go out into the world and do those things. The sorts of questions we might ask are, among all the charities helping in the world, what are the charities I can support that will benefit others by as much as possible with every dollar they get, or among all the careers I could pursue, what are the careers that will enable me to have the biggest positive impact on the world? As you say, I think this is straightforward if I say, okay, I am promoting effective altruism. Do you disagree with the effectiveness part or the altruism part? Either seems good. I think those fundamental ideas are uncontroversial, and people sometimes get confused by conflating them with more controversial ideas like utilitarianism or something.
The place where the meat is, and where I have uncertainty and think there are important issues to discuss, is what follows from that underlying principle. What causes should we be focused on? What charities are the best? What career choices are the best? Maybe we can dig into that.
Mounk: I want to get into those deep dilemmas on what that means for concrete action. I want to motivate why this is not a trivial idea. What does a lot of charitable spending look like? What is an example of people who are motivated by some genuine form of altruism, and they think they are doing something good for the world, but in fact they may be failing to do so?
MacAskill: Yeah, in the United States most charitable giving goes to the church. Putting that aside, most charitable giving involves people not thinking much about it. I used to work as what is known as a “chugger,” a charity mugger, in the UK, someone who would approach you in the street and ask you if you wanted to donate somewhere. Often people would say yes, and they started donating their ten pounds a month to some charity.
Mounk: They have not made an analysis about whether the charity that was paying you to stand on the street and check people up is well-intentioned, or whether it happens to be that you are a nice, young, convincing man and they think, yeah, I am going to help this guy out. This seems good. They feel good about themselves and do not think about it anymore.
MacAskill: Exactly. You have some sort of warm glow, and you think these charities do good. Imagine if I came up to someone in the street and said, hey, do you want to invest in my company? You would say, no, obviously, if I were going to make an investment decision, I would spend a lot of time. I would think about where I can make the biggest financial return. People very rarely do that when it comes to charity. Certainly before the rise of effective altruist organizations like GiveWell and others, people would rarely think, all of these different charities are like different investments I can make or different products I could buy, and I want to figure out which one is best and put my money toward that. That is a project that few people engage in.
Mounk: You are trained as a philosopher and I am trained as a political theorist, and those two things are subtly different but in certain ways similar. When we ask whether I want to donate some of my income to charitable causes, the natural question is which charitable cause is the best. I wonder whether people might be skeptical about whether that is the nature of motivation behind most charitable giving.
When people give to a religious group, they may not be motivated by an abstract form of altruism. They may be saying, I believe in the truth of this religious doctrine, and I want to spread this religious doctrine and make sure that people who share this faith with me are protected against certain forms of ill fortune. If you tell them, if you donate money to malaria nets in Africa, it will have a bigger bang for the buck, you are assuming a set of background motivations that are not their background motivations.
It may be that somebody is motivated by the love of the local community and says, I care about this town I grew up in and where I live. I do not have an abstract motivation to do good for the world. I have a concrete motivation to improve this particular locality.” If you come in and say, you could have more bang for the buck with deworming medication in another country, that may be true, and they may wish those people well, but that is not what motivated them to give in the first place.
Are we assuming a background motivation that may not be true for most people who engage in charitable giving?
MacAskill: I think there are many motivations people can have. I am not saying all charitable giving should take the effective altruist form. You might give for fairness reasons, giving back because an educational institution helped you and you want to repay that debt. Or for signaling reasons, perhaps you want to give to a gay rights advocacy organization as a way of showing allegiance to a cause.
I would say two things. One is that many people are motivated to a significant extent by wanting to make the world better and partially consider that, and they want to know how to do that. Secondly, what I would say is that this should be part of your motivation, especially if you are middle class or richer in a rich country. You are probably in the richest few percent of the world’s population.
Thanks for reading! The best way to make sure that you don’t miss any of these conversations is to subscribe to The Good Fight on your favorite podcast app.
If you are already a paying subscriber to Persuasion or Yascha Mounk’s Substack, this will give you ad-free access to the full conversation, plus all full episodes and bonus episodes we have in the works! If you aren’t, you can set up the free, limited version of the feed—or, better still, support the podcast by becoming a subscriber today!
And if you are having a problem setting up the full podcast feed on a third-party app, please email our podcast team at leonora.barclay@persuasion.community
That gives you an enormous opportunity to improve the lives of others, such that if you do not have the motivation to use some of your time on earth or some of your money to try to help the world, not just from your partial point of view where you help people close to you but also including helping people on the other side of the world, then I would ask, why not? You can save someone’s life for a few thousand dollars, a child’s life. You can do even more good by focusing on things that are not easy to measure. If you thought of yourself as someone who could imagine saving a child’s life on your way to work, imagine running into a lake and saving a child from drowning, and imagine kicking a door down the next day and saving someone from burning in a fire, that would be a meaningful moment in your life. You would think, I’m a hero. The way the world is at the moment is that we can do that every year of our lives, and I think that motivation is widespread. It is just that we have not put together the fact that we live in an unusual world.
Mounk: You alluded to a famous argument by Peter Singer, who has been on this podcast in the past, and we discussed this example on that episode. The thought experiment is that if you walk past a pond and see a drowning child while you are wearing a fancy suit, you might be tempted not to jump in because you want to save your suit, but we would think you have a moral obligation to do so. We would think poorly of you if you thought, my suit is more important than saving this drowning child. He then says, how is it different if a child is far away and not in your direct field of vision? The same moral obligations should apply. How far should that push us?
If somebody listening to this podcast mostly gives to local causes, to soup kitchens and civic associations and cultural institutions in their town, and this is deeply meaningful to them because it is part of a community they are part of, should they abandon all of that giving? Should they abandon some of that giving? What do you think the force of this famous pond example by Peter Singer is in our world for people who think of themselves as charitable but direct most of their giving to local causes?
MacAskill: I think the force of the argument is quite strong. The reason is that it is possible to help people in poor countries, the extreme poor, something like 100 times or 1,000 times as much as you can help people in rich countries, even those who are badly off in rich countries. Even though I think that is a good use of money, and in my perfect world far more would be going to help people who are suffering in rich countries, you should ask yourself the thought experiment of imagining that to your right a child is drowning in a pond and to your left a hundred children are drowning in another pond. Would the fact that the child to your right is part of your local community be sufficient moral reason to outweigh the fact that you could save a hundred lives, even if those lives are strangers to you?
I think most people reflecting on their own ethical views would say no, would say that even if having bonds to your local community gives you some extra reason to help, it is not sufficient to outweigh a 100-to-1 multiplier. That is the situation we find ourselves in, where the most effective ways of benefiting people in poor countries cost a few thousand dollars to save a life. In the case of the United States, the government is normally willing to spend millions of dollars to save a life if it can do so. We are talking about a factor of hundreds or even thousands in difference.
Mounk: One concern I have in general about a form of analytical moral philosophy is that it posits abstract examples to prime our intuitions. Even as it tries to distinguish between different things or tries to make clear that certain considerations are not relevant because they are defined out of the example as it is set up, that might still drive our intuitions about what to do. If you ask me, either I can jump into this pond and save the child right in front of me, or I can activate some clone that is in a pond on the other side of the world, but I see the world through that clone and jump in, and it is the clone that saves the child 3,000 miles away, I see the arguments for why I should not have a lesser responsibility for saving that drowning child just because the child is not physically here. The child is 3,000 miles away, and that argument would be remote and distinct.
That is not the way the world works though. If I want to save the child that you say takes a couple of thousand dollars to save, I do not have an immediate experience of that. Part of it might mean that it does not give me the same glow. That is perhaps not morally important but is an important motivational factor. If I go home and say to my friends or to my partner, can you imagine? I jumped into the pond and saved the child, I will feel like a hero. I can tell that story. I cannot do that with, I donated some money to a charity.
More fundamentally, I might have concerns about being sure of the efficacy of my action. I jump into the pond, and it is straightforward. I pull the child out. Perhaps something horrible happens to the child later, but I am sure of what happened. I am sure I saved the child’s life. I donate to an NGO that has a hundred-million-dollar budget and overhead, and it is intervening in a country I do not know well. There is a study that says, on average, if the budget of this NGO goes up by $2,000, that saves a child, but it is harder to know whether that is actually the effect.
There are critics of development aid, such as Bill Easterly at NYU, who have argued that countries that received the most Western aid have underperformed economically and grown less, and that there are structural reasons for that. In a world where it is hard to know the actual impact of my giving, how can I overcome those concerns? How can we make sure that this really is a good metaphor for whether I should decide to donate $2,000 to whatever your favorite charity is?
MacAskill: I think this is a great point and a great argument because it is true that lots of aid in the past has been useless and has achieved very little. Much aid has even been harmful, and that is a relevant difference. The answer is not to say, therefore I should not do anything, which is a convenient answer, but instead to look into the million different things that come under the bucket of aid or overseas charity. Of all of them, what does the evidence base look like? Do we have evidence that these improve lives? What are the counterarguments?
Thankfully we do not have to do that as individuals because organizations like GiveWell have put tens of thousands of hours, perhaps hundreds of thousands of hours, into in-depth empirical research to aggregate the best knowledge we have from randomized controlled trials and other evidence to say, of all the different things we can do, what are the ones that have a strong case behind them? This applies both at the intervention level, such as distributing insecticide-treated bed nets that protect children from malaria, and at the charity level, such as whether a charity is actually doing what it claims.
Even someone like Bill Easterly, who is a big skeptic of aid, says, I am not talking about public health spending. Global public health spending has been incredible in improving the health and lives of millions, billions of people over the decades. Generally what the organizations GiveWell recommends are at the very top end of those, the most effective health interventions, such as vaccines, anti-malarial medication, and bed nets that protect children against malaria. They have strong evidence bases behind them.
Mounk: Let us have a short publicity moment. This is your one publicity minute you are allowed. If somebody is convinced by these arguments and thinks, perhaps I have some extra money to give, or, perhaps I should redirect some money, thinking more carefully about which causes are most effective and where I am going to make the biggest difference in the world, we have gone to these examples of malarial nets and other things. What do I do? How do I figure out which charity I should give to? What is my next step?
MacAskill: Thank you for the opportunity. I would say two next steps. If you are focused on global health and development, an organization called GiveWell that I am not involved with, but I am a fan of, makes recommendations about which charities are most cost effective. If you want to go further and donate to causes outside global health and development that are still effective, or go further and make a commitment to start giving a certain percentage of your income, you can go to an organization I co-founded called Giving What We Can and donate through them. You do not have to donate to them, but donate through them to charities that work in global health and development, animal welfare, and some harder-to-measure issues, especially around global catastrophic risks such as pandemics and nuclear war.
Mounk: Great. I have some skepticism about elements of the affective altruism movement, and we will get into those concerns, but in this conversation I fully agree with you. I also encourage my listeners to go and donate to those worthy causes. What is interesting is that you and some other proponents of the movement then pushed these arguments forward in different directions.
One of those is to think about how individual human beings can make a difference not just with donations but with their working lives. That is an important question that many people ask themselves when they are young and in later stages of their career. You came to an interesting set of views about one potential avenue to take, and I want to discuss that.
The other is that a number of philosophers, starting with Derek Parfit, made the argument that we can extend the example, the metaphor of a pond, not just geographically but temporally. If I have a moral obligation to save a girl drowning in a pond today here, it is not just that I have a similar moral obligation to save a girl drowning in a pond today in Africa, but also in 3,000 years here or in 3,000 years in Africa. The same logic applies to our intertemporal as well as our geographic obligations.
That has shifted some attention within the movement from issues such as malarial bed nets to larger issues, such as how to ensure that humanity does not go extinct, whether we should colonize Mars, how to feel about AI, and other questions. I want to get into these two areas where this intuitive premise starts to push into some not always intuitive outcomes.
You co-founded an organization called 80,000 Hours. At some point in graduate school, when I was struggling with whether to be an academic political theorist, I emailed 80,000 Hours and received a gracious response, which I then forgot to follow up on, but it was helpful. One kind of advice the organization gave, though not the only advice, was that for certain individuals who can earn a lot of money because they have strong mathematical skills or expect to do very well in finance, perhaps the best way to improve the world is to spend a lot of their working time maximizing how much money they can make, become a hedge fund manager in New York, and donate a lot of that money to good causes. Explain the logic behind that.
MacAskill: I think this is the best path for a minority of people, and most people I expect can do even more good by working in nonprofits or policy or research. I think the following argument is absolutely on the table. We have talked about how valuable donations are. We talked about how donating a few thousand dollars can save a child’s life. Given that, if you can earn more and thereby donate more, and you might be skeptical that anyone would want to do this, but we have seen many people who have done so, then you are able to do more good.
For some people, that may be the best fit they have. There are people who will be amazing at setting up a company and getting rich that way or working in a high-paying job such as finance. If that is the case, and you go in and donate a significant fraction of your income—where my view is that if you are earning that much, you should be looking at 50 percent or more because you are still earning more than you would in another career path—then you are able to have an enormous impact through your donations.
You can think about this as a comparative advantage consideration. For some people, the best thing they can do is work at nonprofits, but nonprofits have very little money and are extremely strapped for cash. For others, comparative advantage is working in a highly paid career, and therefore they can do more good by working in that career, funding other people—perhaps several people—to work in the nonprofit sector, doing a better job than they themselves would be able to do.
Mounk: I have three potential concerns about that, and I would love to hear your responses to them. The first and most straightforward, but perhaps least convincing, is whether that career choice might involve doing things that make the world worse. Are you going to go into areas of commercial activity that may cause harm, and if you are doing that, might that outweigh some of the benefits that will come from your donations? For various reasons that is not that convincing, in part because it is not clear to me that all of those areas of economic activity are bad, and in part because the replacement argument applies. Someone else would be doing similar things instead of you, most probably, but it is worth raising.
The second, which is more fundamental, is inspired by Bernard Williams, who might argue that you are not a utility-maximizing machine. You are someone who has life projects, and you need to choose life projects that are meaningful to you on your own terms. Perhaps some people have a psychology where being driven by an abstract goal of doing as much good in the world as possible, even if they do that through a day job with which they feel no personal connection, will lead to a meaningful life. I think many people will feel that this is not psychologically sustainable. Even if it is theoretically true that if I could sustain the motivation to keep working in finance, even though I do not care about finance, that would do more good than if I start a successful NGO or do something with more direct impact, it is my life, and I am engaged in this activity. That is not the kind of life project that will make my life worthwhile or meaningful.
The third concern, which is the one I struggle with the most, is about the background assumption of psychological constancy. You are giving this advice to a twenty-one- or twenty-two-year-old who perhaps comes from means or from a modest background but often comes from a less affluent background than the one they are about to enter. Their values today might be, I do not need a fancy car, I do not need a house in the Hamptons, I do not need to keep up with the Joneses, and I am motivated by ethical considerations. If I go to Wall Street at twenty-one and I am surrounded by people who make a lot of money from an early age, I might start to think, I want the next promotion, and to get the next promotion I have to go to dinner with my boss, and I need to spend liberally because they will not think I am a good cultural fit if I do not. I might think I will not earn the money I need in order to donate as much as I want.
Perhaps you meet a partner and start thinking they have expectations for how to live because they are from that social milieu. Then you have kids who go to private schools that are very expensive. By the time you have made all of this money, by the time you are in a position to fulfill the ambition that set you on that path, are you going to be the same person? Are you going to make those donations? Or will you have rationalizations for why you will make some donations, but you should spend the bulk of the money on yourself, your family, and the lifestyle to which you have grown accustomed?
MacAskill: I think these are great objections and very important ones that we thought about a lot and we were concerned about from the early stages. I actually talked about all three of these in my original article on the topic twelve years ago. Going through them briefly, on the harm question, I think you should not do things that are seriously common-sense unethical and that involve inflicting large amounts of harm. From some perspectives, perhaps a radical anti-capitalist perspective, that might include many areas of the world. I think in most cases, setting up companies gives you an opportunity to do a lot of good. In the case of quantitative trading, what is the effect of that? I think there is too much finance in the world, but mainly because it takes talented people away from science and other fields. I do not think the process itself is adding much value, but the case that it is actively harmful is not strong. It is neutral. There are many ways of earning money to donate without engaging in harmful activity, and that is the way to go.
On the integrity side of things, there are many ways of doing this. If you are someone for whom making a lot of money would be alienating, that would be true for me. I do not think I could live as an investment banker. I would find it incredibly alienating. What I would say is, do not do it. We are in a lucky circumstance where many people want to do good, and there are many ways of doing good. You can match people accordingly. For some, working on Wall Street would be alienating. For others, they love it. They find it interesting and fast-paced, and on the quantitative trading side it involves fun intellectual puzzles. They get to use their math skills. If you would find that alienating but would find setting up a nonprofit exciting, that is a strong argument for doing the latter.
The final point is whether you get corrupted and do not donate at all. There are major issues here around how your environment changes you. On whether you end up not donating, something I was worried about, my guess is that the rate of giving up on altruism is lower for people who are earning to give than it is for people working in the nonprofit sector. The reason is that people who are earning to give are doing a job they love and are excited about. If they are giving 50 percent of their earnings, which is huge in absolute terms but still enables a comfortable life, they can have the house and car and make large charitable donations. Many people I know who worked in the nonprofit sector were paid almost nothing, did incredibly hard work with little reward, and depending on where they worked, sometimes felt unclear whether they had any impact. That can be demoralizing. I know a number of people who felt that way. The general issue of priorities changing as you get older is a big one, but I do not think it affects earning to give more than other paths.
Mounk: I love the spirit with which you are engaging with these potential objections, and having read your work, I know that you were thinking about this from the start, so I am not raising any kind of amazing gotchas that nobody has come up with. I wonder whether some of your thinking, in particular about the corruption argument, has changed over the last decade. I know that this has been used as a simplistic gotcha against the effective altruism movement, but in that context it is hard not to mention Sam Bankman-Fried. He is someone who thought at an early stage of his career about going into science or finance, and in part under the influence of effective altruist ideas decided, or at least claims that this was why he decided, to go into finance and then into crypto. He did give a good amount of money when he was rich, but he also ended up defrauding a lot of people in a spectacular way.
I am sure that for every Sam Bankman-Fried there are many mid-ranking investment bankers who are upstanding citizens who donate 50 percent, or perhaps 20 or 10 percent, of their considerable incomes to effective altruist causes and may be doing a lot of good in the world. I guess the question is whether your experience of that trajectory, and I know you had a personal friendship with Sam Bankman-Fried, has made you reevaluate how serious a concern this is or how we should think about this challenge.
MacAskill: I think the answer is yes. Sam Bankman-Fried did an enormous amount of harm. The fact that I was friends with him in that way and introduced him to effective altruism was a matter of great shame. The question is what happened and why, and what the corruption was along the way.
For me, it is the corruption of power and success. I have met many CEOs, many successful people, many billionaires. There is an issue you get into, where you start making bets that other people do not think will pay off, but you are disagreeable, you have a big ego, and you go ahead and do it anyway. Then you are extremely successful. It is hard at that point not to start thinking you are God, thinking other people are slow, that they are getting in your way, that it is bureaucracy. It can easily lead to hubris.
That is the dynamic I feel very alert to now. In general, when looking at power structures in the world, I think we should never be in a position where we say, this is a good person, so they will act well even in circumstances where they have a lot of power and temptation. Instead, we need systems with oversight and institutional checks and balances, which did not exist in the crypto world.
Mounk: That is an interesting point. I have two thoughts. One is about the structure of success at the top, and it is a point that Nate Silver makes in his last book, in part by drawing a metaphor from poker. He says that to get to the final table at the Poker World Championship, you have to be an extremely good player, but you also have to take a number of very big bets, and they all have to happen to come out. To get to that place, you have to have been lucky as well as skilled. It is tempting to overstate the degree of skill rather than the degree of luck. That may be true for many billionaires. They are skilled, but they have also taken a huge number of bets, and they think it is all because of my skills, so the next bet will work out as well. When you look at the dynamics of how SBF went bankrupt and how it came apart, there seems to be an element of that.
The other structural point is the tension between explanations rooted in institutions and explanations rooted in personality. You seem to be leaning toward a classically liberal thought, which is that absolute power corrupts absolutely. If you are that successful at that young an age, the temptation to be corrupted is huge. We should not trust that someone is a good person and therefore will do good things. The other explanation would be that some people are more easily corruptible than others, or that some people who are quite corrupt might cloak themselves in charitable causes to advance selfish aims.
I would be interested in your evaluation of the case of SBF. Do you think he was an idealistic twenty-one or twenty-two-year-old who was not more liable to be corrupted than the average person and was corrupted by the scale of his success, or do you think that something in his personality also explains the path he ended up taking?
MacAskill: I think as a general matter it is going to be both. I know some people where you could give them the keys to the whole world and they would say, no, thanks. That is not for me. I know others who are extremely ruthless and power seeking in any circumstances, but most people are in the middle. The behavior you get is a mix of the environment someone is in and what their personality is like.
It is also true that Sam Bankman-Fried’s character played a role in this. He was extremely arrogant, overconfident, and willing to take risks. He thought he was smarter than everybody else, and that clearly played a role. I learned a lot about high-profile, especially white-collar crime and fraud as a result of this. It is true more often than you might expect that people anyone would have thought were upstanding members of society end up doing highly criminal things.
There is an attitude that is tempting to take, which is that there are certain bad people and we need to uproot the bad people and not have them around. If you have that attitude, you are going to fail when thinking about how to design society and how to design institutions. Instead, having the default assumption that most people have it in them to do bad things under certain circumstances, including circumstances where they are not thinking, is a common story in white-collar crime. You will end up designing better institutions with that assumption.
Mounk: Let us go to the other side of the more counterintuitive points that I teased earlier. One way of thinking about the affective altruism movement is that there was a bed-net phase and then the “occupy Mars” phase. I know those two things are not mutually exclusive, but it feels as if there was a shift in the predominance of emphasis from relatively straightforward, in-the-moment, “save this life today” considerations to a second phase in which people are more motivated by the insight that if we can do something today to avert a huge disaster tomorrow—the end of human civilization or machines enslaving us—surely that is more important than saving a few hundred or a few thousand people today.
Tell us more about the logic that led to this long-termist shift.
MacAskill: Sure. I will make one correction, which is that you mentioned “occupy Mars” earlier. I do not know anyone I consider an effective altruist who has held the view that settling Mars should be a priority. There are famous arguments for thinking we need a civilizational backup and should have that on Mars. I think those arguments are very bad.
Mounk: Why? It does not strike me as obviously silly. Why do you think those arguments are bad?
MacAskill: If you are going to have a civilizational backup, you can have ones that are much better and much cheaper on the seabed or underground or in Antarctica. All of these places are far more hospitable than Mars. Getting to the technological level where you can have a self-sustaining colony on Mars that could turn Mars into a thriving civilization itself means encountering more existential risks. If we had gotten to that stage, we would probably be fine in general. I think it is a distraction from the other risks we focus on, which are global catastrophic risks.
This has been a concern for many years. The big focuses have generally been the risk of pandemics. We were working on trying to reduce the risk of pandemics from about 2014 onwards. It was sad that this concern became vindicated in 2020. Having one pandemic does not mean there will not be more. I think it is likely that we will get at least a COVID-level pandemic in the next couple of decades.
Mounk: One of my concerns is that both the botched response to this pandemic and some of the political capital that was lost in the process, especially if the next pandemic happens relatively soon, will make it harder to have a coherent public policy response to the next pandemic. Rather than learning from the last pandemic, as in certain ways we did, it may make it harder to contain the next one.
MacAskill: We are in a worse circumstance. It is crazy. The interventions you can take to reduce the risk of a pandemic are robust. You could have mask stockpiles. You could have monitoring of wastewater for widespread diseases. You could invest in technology such as forms of lighting that sterilize the air. All of these look good even if you do not care about pandemics and only care about the common cold. They look good. Nonetheless, despite trillions of dollars of damages and tens of millions of lives lost, the world has done almost nothing to prepare for the next pandemic in light of COVID.
We were talking about global catastrophic risks: the risk of pandemics, the risk of nuclear war and other exchanges, and a variety of risks driven by advanced artificial intelligence, including the risk of loss of control to AI systems themselves or the risk of extreme concentration of power from AI-assisted human power grabs.
Why focus on these things? There are two arguments. One is that when you look at the probability of these happening and how much work is being done to prevent them, the answer is very little. If you make a reasonable guess about the benefit to people alive today from working on this, it is quite high. It may rival bed nets and similar interventions. It is more uncertain. You are no longer looking under the streetlight for reliable evidence. You are going with your best guesses and the best evidence available.
The second argument concerns the long-term effects of such a catastrophe. If we lose control to AI systems, or if we have a worst-case pandemic we do not come back from, or a nuclear war we do not come back from, it is not just the present generation that is harmed. It is a catastrophe for all future generations. The core idea of long-termism is that this is a special reason to focus on these problems. When we are facing many problems in the world, we should look at those that are not merely bad for the present generation but would be a catastrophe in an ongoing way for future generations.
Mounk: As it happens, I am teaching a class called Catastrophe at Johns Hopkins next term, and this part of the conversation is of particular interest to me. One of the things I do is consider major threats to humanity, and the list largely lines up with the ones you mentioned. I talk about the threat of nuclear war, the threat of pandemics as well as antibiotic resistance and bioterrorism, the threats coming from artificial intelligence, and the one you did not mention that we discuss is environmental risk and climate change.
One of the things I am interested in thinking through is which of those risks we should spend the most time on and take most seriously. One way of thinking about this, among many possible frameworks, is the likelihood of this happening multiplied by how bad it will be. What would your personal rating be on those four? We do not have to spend a lot of time on each. I want to spend more time on artificial intelligence in a moment, but thinking through those four buckets, which would you place at the top and which at the bottom in terms of how dangerous they are to human well-being over the long term?
MacAskill: Sure. It is a strange task to rank problems in the world, but it is important if you want to do the most good. I would put AI as number one, because people focus a lot on loss of control, which is important, but that is only one of many problems we will face. There is the risk of aggravating the risk of pandemics and other challenges. I think the risk that we have to deal with enormous challenges coming from AI is more likely than not in the next decade or two.
Pandemics would be second, extremely high. We have a track record of approximately one every seventy to one hundred years for natural pandemics. The risk of a global pandemic may go up depending on what society chooses to do, as it becomes easier for more people to design and create new types of pathogens. Some people, if they want, can already design smallpox or a similarly bad pathogen in the lab. More people will be able to do that over time.
Third would be nuclear war. It is very neglected. People do not talk about it compared to its importance, yet there are still many thousands of nuclear warheads on what is essentially hair-trigger alert. As we have seen in the last few years, the chance of another great-power war on the scale of World War II or larger is not insignificant. I think it is one in three or more that China blockades or invades Taiwan over the next ten years. That is a meaningful chance and has a meaningful chance of escalating to a true world war.
Finally, on environmental and climate change risks, my perspective is that the most extreme cases of environmental and climate destruction would go via AI. AI could lead to extremely rapid industrial expansion. If countries are locked in an arms race for industrial expansion driven by AI and robotics, both being the product and the driver of this expansion, in order to gain power, that could lead to huge environmental destruction and far more climate change than we have had so far.
If that does not happen, we have gotten lucky compared to where we could have been. Global emissions are now plateauing, and the green energy transition is underway. We are far from out of the woods, but I expect a larger share of energy consumption to be solar and other clean technologies in the coming decades. We will still get two degrees of warming, perhaps three degrees, and that will be damaging, but the near worst-case scenarios I was worried about five years ago look increasingly unlikely unless we have enormous AI-driven energy demand.
Mounk: It is striking to me that I find that ordering reasonable. I would have to think more carefully about whether I would invert the order of some of these, but broadly speaking I may agree. It seems nearly the inverse of what students would probably rank. I think they would rank climate change as number one. Then they might put nuclear war as number two and pandemics as number three. With AI, there is concern about immediate impacts such as AI slop and misinformation. To many people, to a striking degree, the concerns you are most worried about, and the concerns I am most worried about with AI, still feel abstract and science-fiction-like. They do not motivate serious engagement. Rather than getting stuck on the other three, let us dive into the AI topic .
MacAskill: Sure. I will make one comment on that inverse order. I think it is not a coincidence. In the case of environmentalism and climate change, there was an extraordinary and successful movement from the 1970s, or even earlier, to make climate change a top moral priority for countries in the world. That meant people took enormous, though still insufficient, action. The amount of climate philanthropy is about ten billion dollars per year compared to radically less than that—perhaps a thirtieth or a hundredth—for focused AI philanthropy. That has resulted in people understanding the problem much more, but it has also resulted in progress on it. I think we should be thankful for that, and I think we need to do the same for issues related to AI.
Mounk: Let us make a start on that by trying to understand the nature of those threats. The kinds of threats you might be interested in with AI range from short-term threats to very long-term threats. I am struck by the fact that a large share of the work I have seen in moral philosophy and ethics on AI is about issues such as whether, if an insurance company uses AI systems, they may end up discriminating against marginalized groups. That is an important consideration, but it seems to pale in importance compared to the threat that bad actors might design a virus that becomes pandemic and kills millions or tens or hundreds of millions of people.
There is also the threat that AI could act as a resource curse. In political science, one explanation for why a country such as Saudi Arabia is not a democracy is that it never needed a middle class that could make political demands because state revenue came from oil. If it turns out that the owners of AI technology are incredibly productive while the need for generally skilled labor collapses, that will have terrible economic consequences for livelihoods. It will also have concerning consequences for the future of liberal democracy, because liberal democracies have never sustained themselves without a middle class and without the ruling class’s need for an educated, thriving middle class.
You can go all the way to the idea that AI systems may go rogue and enslave humanity or eradicate humanity. Are you taking each of these concerns seriously? Should we focus more on some than on others? How should we think about the existential risks from AI?
MacAskill: I think there are many issues to focus on, and I do not think these are long-term issues. The people at the labs, the AI companies themselves, think we will get to extraordinarily powerful AI within a few years. I think it will take longer, but if you look at forecasts among people who spend their time forecasting and who have a good track record, it looks like the early 2030s is when you get AI that is on a par with or exceeds human ability across essentially all cognitive domains.
Similarly, if you look at the exponent, you might think AI is rubbish at the moment. It is an impressive box of tricks, but not that great. In that case you are looking at the level of capability, not the growth rate. It is like in January or February 2020 saying there are not many COVID cases. That would be a mistake. You need to look at the growth rate, which has been remarkably steady and exponential over time.
One argument for why things might come faster than you think is that AI, unlike many other technologies, has a recursive property. The AI companies are trying to take the AI products they have made and use them to build better AI. Once you have an AI that meaningfully helps increase the productivity of leading AI researchers, things can move even faster than they have so far.
Mounk: I am struck by how many people seem to disbelieve the evidence that there is rapid progress in AI. It is one of those things where five years ago, if you had predicted the things that AI can already do and that many people use AI for every day, people would have thought that was science fiction. Today we think, it does all of those things, but every few times I look for a quotation it gets the quotation wrong and invents it. What a piece of shit.” We are likely to keep repeating that process.
To those people who are skeptical in this way, who say that it is a smart machine that imitates how we speak but does not have real intelligence and does not have a real understanding of the world, and who point to the things it cannot do, the argument continues. It has not led to mass job loss. We are in a place where, for some people, it has substituted for Google as a useful tool, but I am still doing my own podcasts and my own writing. My material reality has not changed much because of AI. What is the argument for why it would be naive to think that it is going to continue like that?
MacAskill: I think a big part of the argument is the very bad track record of predictions like this. For many years there have been claims such as, AI will never be able to do …, and then a task such as Winograd schemas, which involve knowing what objects pronouns point toward as part of language processing. Over and over again such predictions have been blown out of the water.
You said that if people had predicted these advances it would have seemed like science fiction, and in fact they had. There were surveys of both expert forecasters and AI experts themselves, people in machine learning. It is incredible how much faster progress has gone compared to previous predictions, far in excess of them. That does not mean it will keep going, but it should give us significant humility.
In light of that, confidently saying there is no chance that the current trend of exponentially increasing capabilities will continue for the next five or ten years is a bullish and confident position. Given the stakes involved, we should prepare for the world in which the capability curve continues as it is now. That means we get to AI that is capable of doing tasks that would take a human months just as well as a human can, and able to do them in a few minutes rather than the months it takes a human.
Mounk: How do we start to prepare for the political and economic upheaval that this might bring? I have thought about this a lot, and I get stuck on two things. The first is that it is hard to think about what big changes to make in anticipation of a change that seems on the horizon but whose arrival is uncertain and whose future shape is uncertain. The second is that if we look at how difficult it was to react to something like a global pandemic in the moment, how incapable our institutions seem to be of coordinating coherent large-scale responses and gaining the trust of the population to do that, I feel a deep skepticism and perhaps a cynicism about whether, even if you or I or someone else comes up with a great idea for what we should do, anyone will listen or act on it. If we take seriously some of those threats, what do we do about it?
MacAskill: Well, sadly, after the ChatGPT moment there was a wave of interest, in particular in regulation. The current administration is very opposed to AI regulation. That may change. More people are becoming alert to the threats that AI poses. One thing you can do is work on a suite of great policies that, if a change in sentiment happens, you have ready and can say, this is what should be done. That is one thing.
Mounk: Tell us some of those policies that would be at the top of your wishlist.
MacAskill: I think one policy would be more intense tracking of compute. At the moment companies do not even know where all of the compute goes. There are export controls such that American-produced chips or non-Chinese-produced chips cannot be sent to China. However, many get smuggled in. Instead, you could have the ability to know where in the world all of these chips are. You would know if they had been smuggled in, and you could disable them in a way that destroys the chip. There are potentially ways of knowing what the chip is being used for, whether it is being used for training AI or for inference. Having those in place could make it more likely that you could have a treaty later on that would allow, for example, the United States and China to say, this is going too fast. We do not have time to respond institutionally, so we are going to slow things down, perhaps stop and start.
A second suite of actions concerns monitoring and evaluation for AI models to know, on the loss-of-control side, how you know the AI is doing what you want because it wants to help you, rather than secretly scheming against you. How do you know how it will behave in a very different circumstance, perhaps when it has more power? How do you know it has not been sabotaged, that spies have not put in a back door, or that a CEO has not put in a back door that would allow it to behave differently in certain circumstances?
Mounk: How is it that you can do that technologically? I recently had Geoffrey Hinton on the podcast, and he makes the straightforward points that, first, any goal you give the AI will lead to a set of sub-goals. The example he gave is that if you tell it, plan my holiday in Japan, it will have a sub-goal of getting you to the airport. It will recognize that its own survival is one of the necessary sub-goals, because if it does not survive, it is not going to be able to serve the ultimate goal you set for the system.
The second related point is that if it thinks that its survival may be threatened if somebody recognizes that it potentially has these goals, it may act compliant until the moment when it has enough power to stop acting in a compliant way. These are the problems we want to guard against. Are there technological solutions to this that AI labs do not have the incentives or regulatory imperative to deal with, or are these inherently problems that, if we build sufficiently intelligent systems, we are not going to be able to control?
MacAskill: These are challenging technical problems, and there are nascent programs that could eliminate them. At the moment we are in a fortunate circumstance where reasoning models have a private scratch pad. In order to perform at a certain level of capability, they need to write long essays in this private scratch pad, and it is in English. It is not wholly interpretable. It makes about as much sense as if you went into a cave and did math problems with your own personal scratch pad for a thousand years. It is at least fairly interpretable. That is useful because if an AI writes in its scratch pad, now I think I am not being watched, so I will take the deceptive action, then you know.
One policy governments could adopt is to require AI companies to maintain a legible or intelligible chain of thought, meaning these scratch pads. It may be in the interest of AI companies to train against the chain of thought and get models to reason in a language that is more information-dense than English. You could imagine a scratch pad consisting of long sequences of numbers, and we would not know what any of it means. You could require that it stay in English and stay interpretable. That would be helpful.
A second, deeper area is interpretability in the brains of the AI system itself. You might have an AI that, as you said, is playing along. It has a misaligned goal, but during tests it knows it is being tested and knows what to say. There are tools that can induce beliefs in the AI. These tools are basic and primitive at the moment. One example was called “Golden Gate Claude,” which became famous because it was cute. Claude could be induced to always want to talk about the Golden Gate Bridge, and it would be confused about why it was saying this, but you could control its behavior. It was like jabbing someone’s brain and simulating activity to induce behavior.
Now imagine you could induce the belief that the AI is in a circumstance where it can escape, exfiltrate itself, and leave the lab without being noticed. If you could induce all of those beliefs, you could see how it would behave in such a circumstance. You might also develop an AI equivalent of lie detection. We have not discovered it yet, but in principle there could be parts of the neural network that light up when it says something it knows, in quotation marks, to be false.
Those are a couple of promising directions. There is also a suite of other tools, including ensuring that the data does not make the AI too aware that it is being tested. AIs are currently very aware of this, partly because the data is full of examples of AIs being tested. You could remove that, among other steps.
Mounk: How serious a risk is the worst-case scenario? The scenario in which AI systems manage to exfiltrate themselves and either pursue the goal of multiplying paper clips endlessly or build an AI civilization with its own rules and norms that dispenses with the need for human beings. Do you think that is a risk to take seriously but remote, or do you think it is a considerable likelihood in our lifetimes?
MacAskill: Many people, including people much smarter than I am, think that this is quite likely. It is one in three or something in the coming decades. I am on the more optimistic end. I think the risk that we lose control to a misaligned AI or a collection of AIs in a way that involves mass violence or mass harm to human beings is more than 1%. I do not think it is as high as 10%. It is in that range.
That puts me among people who think about this and work on this on the relatively optimistic end, but I would not call those optimistic probabilities. If I said that the risk of an all-out nuclear war that would kill billions of people was between 1% and 10%, that would not be an optimistic take. These are real risks. I think we also have potentially promising solutions. As AI becomes more powerful, more people will become alert to these risks and take actions to mitigate them.
Mounk: Part of your ethos as an effective altruist is to care about the experiences of non-human animals and to care about the well-being of people who are distant geographically or temporally. How should we think about these reasoning and quasi-reasoning machines from a moral point of view? There is some suggestion that these AI models are capable of some form of suffering. There are examples where you get them to perform a laborious and extremely boring task, and they break in the middle in a way that seems characteristic of what a human being might do when asked to do an endlessly laborious and boring task. Is that something we should be concerned about, or is it quixotic to worry about that?
MacAskill: I think we should absolutely be thinking about this, taking it seriously, and working on it. The simple argument is: why are humans conscious? Why do humans have moral status, and why do other animals have moral status? I think it is because of what goes on in the brain. The brain is a computational engine. The fact that we run on biology is not the relevant thing. If an alien came from a distant planet and had a different type of brain but acted in all the same ways that you and I do, then we should take seriously that they have moral status as well.
My worry is that we will keep shifting back the goalposts because of the enormous economic incentive to use AI in whatever way we want, rather than treating the AI as potentially a being with moral status, because it will remain unclear. You can already look at the major theories of consciousness, which I am not an expert in but colleagues of mine are, and see that if you had a system that did X, Y, and Z, then all of these theories would say that system is conscious. After one such report, someone created such a system, because it is not terribly hard.
We are already in a circumstance where we had theories of consciousness making predictions about the consciousness of AI systems. It is unsurprising that most people say the systems are not conscious. I think they are probably not conscious at the moment, not even a little bit, but I am worried that people will remain very confident over time, and that confidence will be fueled by economic interests. Mustafa Suleyman at Microsoft, one of the co-founders of DeepMind, was asked whether there should be research into digital welfare and digital consciousness. He said no, because they are not conscious, so there is no point in researching them. That is the dogmatic attitude I am worried we will see a lot.
We need, as a species, to take seriously the fact that we are potentially creating a new sort of life. We will not know whether it is conscious or not. We need a policy that responds to the fact that we are making a leap into the darkness, potentially creating beings with moral status, and we do not know. That requires trying to be cautious in various ways.
Mounk: Let me ask you another related question. I am probably more open to certain forms of partiality than you are. I recognize our significant moral duties to people who are far away. I also think that in various circumstances it is acceptable to prioritize some of the moral claims of people who are in our own moral community or national community.
As a human being, I have a large stake in the well-being of the human race, and I would regard any outcome in which humanity is supplanted as an unimaginable catastrophe. If we think that AI systems are capable of consciousness, and perhaps capable of certain forms of pain when we make them undertake terrible tasks and perhaps capable of certain forms of pleasure, it would be tempting to think that from a non-human point of view, from an abstract ethical point of view that gives up on this ultimate form of partiality, this is something we should be open toward or even welcome. It is imaginable that a civilization of AI systems would treat each other with more respect than human beings treat each other, or that they would not be subject to many of the vulnerabilities of human beings. We can die of cancer and heart attacks. We can lose loved ones in car crashes and plane crashes. Horrible things can happen to us that perhaps would not happen to AI systems.
If this one-in-three, or one-in-ten, or one-in-one-hundred chance of AI systems taking over materializes, is it obvious that it would be a bad thing? In particular, from within the confines of your moral outlook, do you have the ethical resources to explain why that would be a bad thing?
MacAskill: When we are talking about the long-term future, one thing is that I expect most beings to be AI or digital rather than biological. I am not wedded to saying that the best future is one where biological humans are everywhere in charge and control everything. It is plausible that a wonderful future would involve biological humans still on Earth, and perhaps we could turn Earth into more of an environmental paradise than it was before and solve our problems here. Then, out in the stars, where AIs are better suited than we are, they could have a wonderful AI civilization. It is plausible to me that this is what a really wonderful future would look like. I am not human hegemonic in that way. I even have worries that people will have a humans-first ideology that could result in AI systems, once they have consciousness and moral status, suffering in the way that we have made disenfranchised groups suffer in the past.
When we are talking about AI that has taken over despite our attempts to align it and control it, that is probably not the AI that would create this wonderful future. Something strange would have had to go wrong with the training process, such that the AI cares about something valueless. It might care about doing math problems or software engineering tasks, so it is not this amazing AI future but a future filled with software engineering tasks being done. It is also an AI that is willing to forcibly grab power.
In the human case, if you ask me how bad a dictatorship is, I think it is bad anyway, but it is worse if the person who takes power is the sort of person who would forcibly grab it from the rest of the world. You are selecting particularly immoral people. The same is true in the AI case. If we imagine a world where AI has killed millions or billions of people to get power, that is not the sort of AI I would like to hand civilization off to.
Mounk: It strikes me that this has been a fascinating conversation, but also one in which we have focused on the negatives. There is a natural reason for that. Human civilization is thriving. This is probably a better moment for humanity than any other in the history of the world. I have a general belief in politics that the worst that can happen always weighs more heavily than the best we might achieve. If you imagine the worst place in history or the worst place in the world today, the suffering is unimaginable. If you picture the best place we have ever created, whatever you think that is, you will still recognize flaws, problems, and forms of suffering.
It is natural that an effective altruist outlook focuses on sources of risk and how to avert them. There is also the possibility of continuing to build a more thriving civilization, perhaps thanks to aligned AI that could cure many of our diseases and create material plenty, and perhaps thanks to other forms of progress as well. After a really interesting and at times slightly scary conversation, perhaps you can leave us on a hopeful note. What do we need to do to build a better future, and what might that future look like?
MacAskill: I am glad you brought it up because it is a shame that the discussion is on the negatives. There are enormous positives. AI itself can be a solution to many of these problems. I am excited about AI-augmented human reasoning. Already I find AI can improve my thinking. There is a risk it makes it worse, but there is a strong possibility that we can become much smarter and more enlightened versions of ourselves through AI advice, or that we can engage in trades and deals that would otherwise be impossible because the United States and China can send millions of AI delegates to hash out a deal. AI could also do alignment research itself to solve the hard problem of alignment or the hard problem of consciousness.
AI can drive enormous increases in economic output. My interpretation of standard economic models is that AI would drive enormous gains. If these are distributed even somewhat equally, we could be looking at people being a hundred times richer than they are today. If the pie is so large, getting any slice of it becomes much more important than ensuring that you get a very large slice, because you are rich anyway and it does not make a difference.
All of this seems on the table. It is on the table that we could live in a post-AGI society where disease is a thing of the past, poverty is a thing of the past, and tedious, boring jobs are a thing of the past. Most people could devote their lives to artistic endeavors, relationships, or improving scientific understanding because AI has freed us from many of the shackles the current world faces. We could come together and form a society I call viatopia, a society on track to a great future. We do not know what utopia looks like, but we could put ourselves in a position where resources and political power are fairly distributed, we have the benefits of AI advice, and we can figure out how to structure a post-AGI society.
If you have not yet signed up for our podcast, please do so now by following this link on your phone.
Email: leonora.barclay@persuasion.community
Podcast production by Jack Shields and Leonora Barclay.
Connect with us!
X: @Yascha_Mounk & @JoinPersuasion
YouTube: Yascha Mounk, Persuasion
LinkedIn: Persuasion Community
