There is an easy way to see if a human is conscious in this way: ask them. When it comes to AI systems, things are different. We can’t simply ask large language models like ChatGPT and Claude whether they are conscious. Robert Long (2023) explores the extent to which future systems might provide reliable self-reports. 

LLMs are not just “stochastic parrots”

Well, we can ask.

As an AI, I do not have the capability for self-awareness or subjective experience. Therefore, I cannot truly “know” or report on my own consciousness or lack thereof. My responses are generated based on patterns and data, not personal reflection or awareness. 

This reflects a view that AI systems are stochastic parrots. On this view AI models:

• Exploit mere statistical patterns in data, rather than invoking concepts to reason.

• Rely on information from training rather than introspection to answer questions. 

Arguably, both points are not true. 

LLMs can distinguish between facts and common misunderstandings. This indicates an ability to distinguish between more than mere statistical relationships in text (Meng et al., 2022)

AI systems seem to have limited forms of introspection. Introspection involves representing one’s own current mental states so that these representation can be used by the person (Kammerer and Frankish). There is preliminary evidence that AI systems can introspect at least a little. For instance, some models are able to produce well calibrated probabilities of how likely is that they have given a correct answer (Kadavath et al., 2022). However, it has not been established that these representations are actually available to the AI models when answering other questions.

AI systems seem to have concepts. One piece of evidence that they are very capable of using compound phrases like “house party” (very different from a “party house”). This indicates that they are not doing something simple like averaging statistical usage of individual words. 

Indeed, we can look inside AI systems to find out. Since Robert Long published this paper last year, there has been a lot of progress applying mechanistic interpretability techniques to LLMs. A recent Anthropic paper showed that their language model had intuitive concepts for the Golden Gate Bridge, computer bugs, and thousands of other things. These concepts are used in model behaviour and understanding them helps us understand what the LLM is doing internally. 

Some researchers have also found structural similarities between internal representations and the world (Abdou et al., 2021; Patel and Pavlick, 2022; Li et al., 2022; Singh et al., 2023).

Why do LLMs say they are (not) conscious?

LLMs are initially trained to predict the next word on vast amounts of internet text. LLMs with only this training tend to say that they are conscious, and readily give descriptions of their current experiences. 

The most likely explanation of this behaviour is that claiming to be conscious is a natural way to continue the conversation as humans would, or as dialogue in science fiction normally goes.

Recently, consumer facing chatbots have stopped claiming to be conscious. This is likely due to the system prompt, which is given to the LLM but not shown to the end-user. For example, one model was told ‘You must refuse to discuss life, existence or sentience’ (von Hagen, 2023). System prompts are part of the text that AI systems are generating a continuation of, so this likely explains why current consumer chatbots no longer claim that they are sentient.

Could we build an AI system that reflects on its own mind?

Early on, people thought that any AI system that could use terms like “consciousness” fluently would, by default, give trustworthy self-reports about their own consciousness (Dennett, 1994). Current models can flexibly and reliably talk about consciousness, yet there is little reason to trust them when they say they are conscious. However, given that LLMs have some ability to reflect on their own thinking, we may be able to train systems that give accurate self-reports.

The basic proposal is to train AI systems on questions that involve introspection that we can verify for ourselves, then ask about more difficult questions such as whether and what the AI systems are thinking. Here are some suggested questions adapted from the paper:

An AI system trained to introspect on its own internal states that says it is conscious may give us evidence that it is conscious, in a similar way to how humans give us evidence about how they are conscious. 

One interesting thing that comes out of this approach is the idea of a training/testing trade-off. This trade-off is between giving an AI enough context to understand and competently deploy concepts such as “consciousness” on the one hand, avoiding giving the system too much information such that it is incentivised  to extrapolate from the training data rather than introspect. This may be hard to achieve in practice (Udell and Schwitzgebel, 2021). 

My own thoughts on the paper

• I like this paper, it is an interesting approach to understanding AI consciousness. It improves on previous work such as Turner and Schneider’s (2018) test of AI consciousness. 

• While I have concerns about the methodology, I do think that if this kind of investigation was done and interpreted extremely carefully by competent researchers, then it would give us crucial evidence about AI consciousness.

• At the same time, it seems clear that we need other approaches, which I plan to summarise on the blog soon. In my view, the most compelling approach: uses our best neuroscientific theories of consciousness to create a list of indicators of consciousness, that we can check for in current and future systems. These theories are not perfect, but if a future system meets the standards for consciousness across many plausible theories this seems about the strongest single piece of evidence that we could have for AI consciousness.

• I wonder if teaching self-reflection changes whether or not an AI system is actually conscious. For instance, could it be that LLMs are not conscious unless they are trained to self-reflect? Alternatively, could it be that this training changes their experiences in morally significant ways? I’d like to investigate this in the future (I’d love to talk to anyone thinking about this in depth).

• Perez and Long (2023) give a more detailed description of how to build an introspective AI system that might give useful answers to questions about consciousness and moral status. I haven’t read this yet.

  Share with someone who can introspect better than an AI in 2024

  Share

  Cover image: "The first conscious AI system" as imagined by Stable Diffusion 3 Medium.

This is a summary of  “The scope of longtermism” by David Thorstad (forthcoming in the Australasian Journal of Philosophy). The summary was first published by the Global Priorities Institute. 

Rapid diminution

We cannot know the details of the future. Picture the effects of your actions rippling out in time – at closer times, the possibilities are clearer. As our prediction journeys further, the details become obscured. Although the probability of desired effects becomes ever lower, the effects might grow larger. In the long run, we could perhaps improve many billions or trillions of lives. When we weight value by probability, the value of our actions will depend on a race between diminishing probabilities and growing possible impact. If the value increases faster than probabilities fall, the expected values of the action might be vast. Alternatively, if the chance we have such large effects falls dramatically compared to the increase in value, the expected value of improving the future might be quite modest.

Thorstad suggests that the latter of these effects dominates, so we should believe we have little chance of making an enormous difference. Consider a huge event that would be likely to change the lives of people in your city – perhaps, your city being blown up. Surprisingly, even this might not have large long-run impacts. Studies indicate that just half a century after cities in Japan and Vietnam were bombed, there was no longer any detectable effect on population size, poverty rates and consumption patterns.1 To be fair, some studies indicate that some events have long-term effects,2 but Thorstad thinks ‘...the persistence literature may not provide strong support’ to longtermism. There are few established examples of events with persistent long-term effects, there are sometimes alternative explanations for these persistent effects, and these examples tend to be events with large short-term effects as well.3

Washing out

Thorstad’s second concern with longtermism relates to our ability to predict the future. If our actions can affect the future in a huge way, these effects could be wonderful or terrible. They will also be very difficult to predict. The possibility that our acts will be enormously

beneficial does not make our acts particularly appealing when they might be equally terrible. If our ability to forecast long-term outcomes is limited, the potential positive and negative values would wash out in expectation.

Thorstad identifies three reasons to doubt our ability to forecast the long term. First, we have no track record of making predictions at the timescale of centuries or millennia. Our ability to predict only 20-30 years into the future is not great – and things get more difficult when we try to glimpse the further future. Second, economists, risk analysts and forecasting practitioners doubt our ability to make long-term predictions and often refuse to make them.4 Third, we want to forecast how valuable our actions are over the long run. But value is a particularly difficult target – it includes many variables such as the number of people alive, their health, longevity, education and social inclusion. That said, we sometimes have some evidence, and this evidence might point to an act that seems to be slightly more likely to improve the future than ruin it. Even then, our situation is bleak. We only observe a small amount of the evidence that weighs on these issues, and this evidence may mislead. Instead of taking the evidence at face value, we might take it to show that we missed a piece of evidence that would have told us the very same act could devastate the future. Whenever our evidence points towards a specific option that we think will be best for the long term, we may be sceptical that it points the right way.

Option unawareness

Usually, we think of a decision as having a few obvious choices (continue reading, take a break, etc.). This is a simplification. In practice, we often have many options that go unseen (throw shoes out of the window, wear socks on hands, etc.). Longtermism claims that our very best options will be the ones which have the best long-term effects. But in many situations, even if we have an option that has predictably helpful long-term consequences, we may not consider this option while deciding. Also, if we restrict our choice to only those actions we are aware of, we may not readily identify an option with particularly good (or bad) long-term effects. In this way, longtermism might be true for most choices in theory, but false for most of the choices that we actually make in practice.

Conclusion

Overall, these three considerations diminish the scope of the decision situations where longtermism is relevant. In practice, our best options will often be those with the best short-term effects. Although there may be some decisions for which the best option will be one that has enormously good long-term consequences,5 these will be rare exceptions.6

Image by Kama Tulkibayeva.

Indeed, according to Bostrom (2003) if there are many more simulated than non-simulated people, then we are most likely simulated. However, this argument can only ever be as strong as our belief that there actually are many more simulated than non-simulated people. For example, Bostrom believes there is only about a one-third probability that there really are many more simulated people than non-simulated people, so this argument is compatible with a two-thirds probability that we are not in a simulation. In “Simulation Expectation”, Teruji Thomas develops a novel argument for the claim that we are living in a simulation.

This is a summary of the Global Priorities Institute Working Paper "Simulation Expectation" by Teruji Thomas. It first appeared on the Global Priorities Institute website.

The argument for the simulation hypothesis

Thomas’ argument relies on the concept of a ‘reference class’. To illustrate, suppose I smoke, and I want to know the probability that I’ll develop lung cancer. If I knew that one out of fifteen smokers develop lung cancer, I could use my reference class (people who smoke) to make a prediction (1/15 probability of developing lung cancer). In general, a reference class is a way of assigning an initial probability to ‘things like this’.

Thomas applies this kind of reasoning to our probability of being in a simulation. He suggests that we use the reference class of people who inhabit ‘Earthy’ worlds: that is, people who inhabit minor variations of twenty-first-century earth.

The argument goes like this:

Premise 1: Suppose that we ourselves are non-simulated. Then the expected ratio of simulated to non-simulated people living in Earthy worlds is very high.

Premise 2: The fact that we are living in an Earthy world includes almost all the relevant evidence we have.

Premise 3: Given 1 and 2, the probability that we are living in a simulation is also very high.

Conclusion: Therefore, the probability that we are living in a simulation is very high.

Premise 1 notes that, even though it may be unlikely that our descendants are ever able to create sophisticated simulations of the 21st century, if they do, then they may well create a very large number of simulated worlds broadly like our own. Thus, if we are not simulations (so that base-reality is an Earthy world), then the expected ratio of simulated to non-simulated people in Earthy worlds is very high.

Premise 2 is important because additional evidence could change the resulting probability. In our first example, if a genetic test showed that I had a gene associated with increased lung cancer risk, then the probability that I develop lung cancer would be greater than 1/15. In this case, the fact that we live in a minor variation of twenty-first-century earth includes almost all of the evidence we have about whether or not we live in a simulation. It leaves out personal details such as what I ate for breakfast or the colour of my socks, but these details are almost entirely irrelevant to the question of whether or not we live in a simulation.

The paper contains a mathematical proof of Premise 3. The upshot is that the expected ratio of simulated to non-simulated people in our reference class provides a lower bound to the odds that we live in a simulation. Together, these premises support the conclusion that the probability that we are in a simulation is very high indeed.

Carefully interpret evidence

There is a potential objection to Premises 1 and 2: perhaps we cannot define “Earthy” in a way that includes almost all of the relevant evidence (Premise 2) while also ensuring that the expected ratio of simulated to non-simulated people is high (Premise 1).

Consider the (apparent) fact that our world is more than ten billion years old and was mostly lifeless for many of those billions of years. Even if our descendants create many simulations of our world, it is unlikely that many of those simulated worlds will themselves contain vast empty stretches for billions of years; they would just appear that way.  So, if we include this fact in what it means to be an “Earthy” world, then Premise 1 may well fail. If, however, we exclude this fact from what it means to be an “Earthy” world, then Premise 2 may well fail, because we are excluding an important piece of evidence. One possible way to resolve this is to try to understand our evidence, not in terms of how the world is, but in terms of how the world appears to be. Potentially, we can keep both Premises 1 and 2 if the definition of “Earthy” includes the fact that the world appears to be old and mostly lifeless but doesn’t require the world to actually be that way. This raises some complicated issues.

Because of these considerations, we must carefully interpret any evidence we encounter. Imagine stumbling upon a lab running billions of simulations. Would your new evidence make it likely that you live in a simulation? Bostrom (2006, p. 9) and Greene (2020) think so, but the situation is unclear. If your new evidence is only that the lab appears to be running billions of simulations, then it is unclear which way the new evidence points. On the other hand, most simulations probably do not house their own simulations––so if your new evidence is that the lab really is running billions of simulations, then that could make it less likely that you are in a simulation.1

Photo by Tima Miroshnichenko.

The singularity is a hypothetical future event in which machines rapidly become significantly smarter than humans. The idea is that we might invent an artificial intelligence (AI) system that can improve itself. After a single round of self-improvement, that system would be better equipped to improve itself than before. This process might repeat many times, and each time the AI system would become more capable and better equipped to improve itself even further. At the end of this (perhaps very rapid) process, the AI system could be much smarter than the average human. Philosophers and computer scientists have thought we should take the possibility of a singularity seriously (Solomonoff 1985, Good 1996, Chalmers 2010, Bostrom 2014, Russell 2019). 

It is characteristic of the singularity hypothesis that AI will take years or months at the most to become many times more intelligent than even the most intelligent human.1 Such extraordinary claims require extraordinary evidence. In the paper “Against the singularity hypothesis”, David Thorstad claims that we do not have enough evidence to justify the belief in the singularity hypothesis, and we should consider it unlikely unless stronger evidence emerges.

Reasons to think the singularity is unlikely

Thorstad is sceptical that machine intelligence can grow quickly enough to justify the singularity hypothesis. He gives several reasons for this.

Low-hanging fruit. Innovative ideas and technological improvements tend to become more difficult over time. For example, consider “Moore’s law”, which is (roughly) the observation that hardware capacities double every two years. Between 1971 and 2014 Moore’s law was maintained only with an astronomical increase in the amount of capital and labour invested into semiconductor research (Bloom et al. 2020). In fact, according to one leading estimate, there was an eighteen-fold drop in productivity over this period. While some features of future AI systems will allow them to increase the rate of progress compared to human scientists and engineers, they are still likely to experience diminishing returns as the easiest discoveries have already been made and only more difficult ideas are left. 

Bottlenecks. AI progress relies on improvements in search, computation, storage and so on (each of these areas breaks down into many subcomponents). Progress could be slowed down by any of these subcomponents: if any of these are difficult to speed up, then AI progress will be much slower than we would naively expect. The classic metaphor here concerns the speed a liquid can exit a bottle, which is rate-limited by the narrow space near the opening. AI systems may run into bottlenecks if any essential components cannot be improved quickly (see Aghion et al., 2019).

Constraints. Resource and physical constraints may also limit the rate of progress. To take an analogy, Moore's law gets more difficult to maintain because it is expensive, physically difficult and energy-intensive to cram ever more transistors in the same space. Here we might expect progress to eventually slow as physical and financial constraints provide ever greater barriers to maintaining progress.

Sublinear growth. How do improvements in hardware translate to intelligence growth? Thompson and colleagues (2022) find that exponential hardware improvements translate to linear gains in performance on problems such as Chess, Go, protein folding, weather prediction and the modelling of underground oil reservoirs. Over the past 50 years, transistors in our best circuits increased from 3,500 in 1972 to 114 billion in 2022. If intelligence grew linearly with transistor density computers would have become 33 million times more intelligent over this period. Instead, evidence suggests that intelligence growth is sublinear in hardware growth. 

Arguments for the singularity hypothesis

Two key arguments have been given in favour of the singularity hypothesis. Thorstad analyses them and finds that they are not particularly strong. 

Observational argument. Chalmers (2010) argues for the singularity hypothesis based on the proportionality thesis: that increases in intelligence always lead to at least proportionate increases in the ability to design intelligent systems. He supports this only briefly, observing that, for example, a small difference in design capability between Alan Turing and the average human led to a large difference in the ability of the systems they were able to design (the computer vs hardly anything of importance). The main problem with this argument is that it is local rather than global: It gives evidence that there are points in time where the proportionality thesis is correct, while to support the singularity hypothesis it would be necessary that the proportionality thesis is true at any time. In addition, Chalmers conflates design capabilities and intelligence.2 Overall, Thorstad concludes that Chalmers's argument fails and the observational argument does not vindicate the singularity hypothesis

Optimisation power argument. Bostrom (2014) claims that there will be a large amount of quality-weighted design effort applied to improving artificial systems, which will result in large increases in intelligence. He gives a rich and varied series of examples to support this claim. However, Thorstad finds that many of these examples are just plausible descriptions of artificial intelligence improving rapidly, not evidence that this will happen. Other examples end up being restatements of the singularity hypothesis (for example, that we could be only a single leap of software insight from an intelligence explosion). Thorstad is sceptical that these restatements provide any evidence at all for the singularity hypothesis.

One of the core parts of the argument is initially promising but relies on a misunderstanding. Bostrom claims that roughly constant design effort has historically led to systems doubling their capacity every 18 months. If this were true, then boosting a system's intelligence could allow it to design a new system with even greater intelligence, where that second boost is even bigger than the first. This would allow intelligence to increase ever quicker. But, as discussed above, it was increasing design efforts that led to this improvement in hardware, and AI systems have progressed much more slowly. Overall, Thorstad remains sceptical that Bostrom has given any strong evidence or argument in favour of the singularity hypothesis.

Implications for longtermism and AI Safety

The singularity hypothesis implies that the world will be quickly transformed in the future. This idea is used by Bostrom (2012, 2014) and Yudkowsky (2013) to argue that advances in AI could threaten human extinction or permanently and drastically destroy humanity's potential for future development. Increased scepticism about the singularity hypothesis might naturally lead to increased scepticism about their conclusion: that we should be particularly concerned about existential risk from artificial intelligence. This may also have implications for longtermism which uses existential risk mitigation (and AI risk mitigation in particular) as part of the central example of a longtermist intervention - at least insofar as this concern is driven by something like the above argument by Bostrom and Yudkowsky.3

We might hope that there is a straightforward way of predicting the behaviour of future artificial intelligence (AI) systems. Some have suggested that AI will maximise expected utility, because anything else would allow them to accept a series of trades that result in a guaranteed loss of something valuable (Omohundro, 2008). Indeed, we would be able to predict AI behaviour if the following claims were true:

1. AI will avoid exploitation

2. Avoiding exploitation means maximising expected utility

3. We are able to predict the behaviour of agents that maximise expected utility

Adam Bales argues that these claims are all false in his paper Will AI avoid exploitation?

AI won't avoid exploitation

Here, “exploitation” is meant in a technical sense. An agent is exploitable if you can offer them a series of choices that lead to a guaranteed loss. For instance, if an agent is willing to pay $1 to swap an apple for an orange, but also willing to pay $1 to swap back to the orange, that agent is exploitable. After two trades they would be back with the apple they started with, minus $2. A natural assumption is that AI systems will be deployed in a competitive environment that will force them to avoid exploitable preferences given sufficient training data and computational resources to do so.

But there are reasons to think that AI will be exploitable in at least some scenarios. We might suspect as much when we notice that companies do not avoid exploitation, despite their competitive environment.1 Additionally, animal behaviour tends to be more consistent than human behaviour, despite our cognitive sophistication (Stanovich, 2013). In general, there might be a trade-off between understanding and processing complex information, and choosing in ways that avoid exploitation (Thorstad, forthcoming).

It also seems that the benefits of avoiding exploitation might be lower than they appear. First, there is no benefit to avoiding hypothetical exploitation. AI systems that always avoid exploitation will spend a lot of resources on avoiding hypothetical situations that are unlikely to arise in the real world. Second, even agents who are immune to exploitation in the technical sense may be vulnerable to other kinds of, well, exploitation. To see this, consider how a human might be exploited by someone with better knowledge of the stock market. This kind of exploitation would not rely on patterns of preferences but on a lack of knowledge and ability in a particular domain. So avoiding exploitation (in the technical sense) is less important than it initially sounds. The costs are also higher than they first appear: for instance, avoiding exploitation in a sufficiently general way is computationally intractable (van Rooij et al., 2018).

Even if AI avoids exploitation, it may not maximise expected utility

An agent maximises expected utility if it makes choices that are consistent with the maximisation of the expectation of some utility function. This does not mean an AI system needs to have explicit utility and probability functions, this is just a way for us to understand and predict which decisions it will make.

We can break down the concept of maximising expected utility into more simple behavioural patterns. If an agent satisfies all of the patterns, they maximise expected utility. But if they violate one of these patterns, then they do not. We can use this breakdown to see whether agents that avoid exploitation (in the technical sense) will really maximise expected utility.

In particular, one of the behavioural patterns we expect from an agent maximising expected utility would be "continuity" (Fishburn, 1970). Suppose you prefer outcomes A to B to C but there is also a lottery that gives you A with a probability p and C with probability 1-p. Continuity implies that there is some (high enough)  p  that you would choose this lottery over getting B for sure, and for some (small enough) p’ for which you will choose B.

An agent who violates continuity will not maximise expected utility. However, if an agent avoids exploitation, this doesn’t tell us that they will satisfy continuity. This is because even agents that do not satisfy continuity may still avoid any guaranteed loss (and therefore avoid exploitability). This means that if we know an agent avoids exploitation, we would not know that they maximise expected utility.2

Interestingly, failing to satisfy continuity could result in some strange behaviour. Bales finds that an agent that does not satisfy continuity would pay a cost for a very small chance of getting something better, no matter how small that chance is. Bales calls this quasi-exploitability, but notices that we do not have particularly strong reasons to believe AI will avoid quasi-exploitability, which may even be seen as appropriate under conditions of very high potential payoffs. Ultimately, this line of reasoning fails to show that AI will maximise expected utility.

Even if AI will maximise expected utility, knowing this will not help us predict its behaviour

Even if AI systems did act as if they maximise expected utility, this would not allow us to predict their behaviour. This is because we would only know what they would do relative to some probability function and some utility function. Consider an AI whose utility function assigns a value of 0 to all of the outcomes other than the one it expects to receive by acting the way it does. This agent could be seen as maximising expected utility, but, we will not know which outcomes the agent will choose before it acts.

We might be able to partially get around this by using our knowledge of how the AI is being trained to predict what its utility function would be. In particular, we might assume that future AI systems will be trained in ways that are similar to how current cutting-edge models are trained. However, then we would be making substantial assumptions about what future AI systems will look like. We might wonder if these substantive assumptions now drive the predictions, rendering the utility maximisation framework inert. Additionally, insofar as speculating about how future AI systems will behave is difficult, we might doubt that this approach will give us particularly fruitful insights.

Conclusion

Overall, the failure of these three claims means that any argument leading to the conclusion that we might be able to predict the behaviour of AI systems would need to be more sophisticated. In particular contexts, AI might approximately avoid exploitation – for example, when it is likely to be exploited, or in its interactions with humans and other agents. When combined with further assumptions about behaviour that might come from our understanding of the training processes that will generate advanced AI systems, we might be able to get some idea of how AI systems will behave. We should be modest in our predictions though, because our assumptions are often likely to miss important insights, oversimplify, or even mislead us.

Photo by Pavel Danilyuk

Sources

Peter C. Fishburn (1970). Utility theory for decision making. Wiley.

Peter C. Fishburn (1971). A study of lexicographic expected utility. Management Science 17/11, pages 672–678.

Hausner (1953). Multidimensional Utility. Rand Corporation No. 604151.

Hausner, M., & Wendel, J. G. (1952). Ordered Vector Spaces. Proceedings of the American Mathematical Society 3/6, pages 977–982.

David McCarthy, Kalle Mikkola, and Teruji Thomas (2020). Utilitarianism with and without expected utility. Journal of Mathematical Economics 87, pages  77–113.

Stephen M. Omohundro (2008). The Basic AI Drives. Proceedings of the 2008 conference on Artificial General Intelligence. IOS Press. Edited by Pei Wang, Ben Goertzel and Stan Franklin.

Keith E. Stanovich (2013). Why humans are (sometimes) less rational than other animals: Cognitive complexity and the axioms of rational choice. Thinking & Reasoning 19/1, pages 1–26.

David Thorstad (forthcoming). The accuracy-coherence tradeoff in cognition. The British Journal for the Philosophy of Science.

Iris van Rooij, Cory D. Wright, Johan Kwisthout and Todd Wareham (2018). Rational analysis, intractability, and the prospects of ’as if’-explanations. Synthese 195/2, pages 491–510.

Political decisions can have lasting effects on the lives and wellbeing of future generations. Yet political institutions tend to make short-term decisions with only the current generation – or even just the current election cycle – in mind. In “longtermist institutional reform”, Tyler M. John and William MacAskill identify the causes of short-termism in government and give four recommendations for how institutions could be improved. These are the creation of in-government research institutes, a futures assembly, posterity impact statements and – more radically – an ‘upper house’ representing future generations.

Causes of short-termism

John and MacAskill discuss three main causes of short-termism. Firstly, politicians may not care about the long term. This may be because they discount the value of future generations, or simply because it is easy to ignore the effects of policies that are not experienced here and now. Secondly, even if politicians are motivated by concern for future generations, it may be difficult to know the long-term effects of different policies. Finally, even motivated and knowledgeable actors might face structural barriers to implementing long-term focussed policies – for instance, these policies might sometimes appear worse in the short-term and reduce a candidate's chances of re-election.

Suggested reforms

In-government research institutes

The first suggested reform is the creation of in-government research institutes that could independently analyse long-term trends, estimate expected long-term impacts of policy and identify matters of long-term importance. These institutes could help fight short-termism by identifying the likely future impacts of policies, making these impacts vivid, and documenting how our leaders are affecting the future. They should also be designed to resist the political incentives that drive short-termism elsewhere. For instance, they could be functionally independent from the government, hire without input from politicians, and be flexible enough to prioritise the most important issues for the future. To ensure their advice is not ignored, the government should be required to read and respond to their recommendations.

Futures assembly

The futures assembly would be a permanent  citizens’ assembly which seeks to represent the interests of future generations and give dedicated policy time to issues of importance for the long-term. Several examples already exist where similar citizens’ assemblies have helped create consensus on matters of great uncertainty and controversy, enabling timely government action.1 In-government research institutes excel at producing high quality information, but lack legitimacy. In contrast, a citizens’ assembly like this one could be composed of randomly selected citizens that are statistically representative of the general population. John and MacAskill believe this representativeness brings political force –politicians who ignore the assembly put their reputations at risk. We can design futures assemblies to avoid the incentive structures that result in short-termism – such as election cycles, party interests and campaign financing. Members should be empowered to call upon experts, and their terms should be long enough to build expertise but short enough to avoid problems like interest group capture – perhaps two years. They should also be empowered to set their own agenda and publicly disseminate their results.

Posterity impact statements

Requiring posterity impact statements for legislation would provide another mechanism for creating political accountability and gathering high quality information on long-run policy effects. These statements would give an estimate of the expected impact of a policy on future generations, similar to the environmental impact statements that are already required in many countries. Posterity impact statements might utilise a “soft” enforcement mechanism – relying on voters to enforce good long-term policy creation – or a “hard” enforcement mechanism – for example, the government might have to take out insurance when implementing particularly risky policies. 

Future generations ‘upper house’

A more radical reform would be to introduce an ‘upper house’ that represents future generations explicitly, to work with a lower house representing current generations. (Legislation would have to pass both houses). John and MacAskill suggest several things that might help such a proposal work:

• Randomly selecting citizens and experts to serve in the upper house (to avoid the incentives that drive short-termism, such as election cycles, party interests, industry corruption and partisan polarisation). 

• Independent research institutions should create concrete performance metrics and members of the house should give public justifications that refer to those metrics. 

• The members should be relatively young, and given a pension a number of decades later based on their cohort’s performance in promoting the interests of future generations. 

Further ideas for reforms

John and MacAskill also suggest several additional ideas for reforms that might be worth exploring further: longer election cycles, novel commitment mechanisms, giving parents additional votes to use on behalf of their children, taxation of negative and subsidy of positive long-run externalities, and long-term performance incentive schemes such as tying the pensions of politicians and public servants to national performance.

References

James S Fishkin and Robert C Luskin (2005). Experimenting with a democratic ideal: Deliberative polling and public opinion. Acta Politica 40/3.

James S. Fishkin, Roy William Mayega, Lynn Atuyambe, Nathan Tumuhamye, Julius Ssentongo, Alice Siu and William Bazeyo (2017). Applying deliberative democracy in Africa: Uganda’s first deliberative polls. Daedalus 146/3.

Tyler M. John and William MacAskill (2021). Longtermist institutional reform. The Long View: Essays on Policy, Philanthropy, and the Long-Term Future. FIRST. Edited by Natalie Cargill and Tyler M. John.

Christian List, Robert C. Luskin, James S. Fishkin, and Iain McLean (2013). Deliberation, single-peakedness, and the possibility of meaningful democracy: Evidence from deliberative polls. The Journal of Politics 75/1.

Photo by Max Avans.

More likely than not, we will see advanced AI systems within our lifetimes, or our children's lifetimes

Carlsmith builds his case for AI risk around estimates for when AI systems could be developed that have: 

• Advanced capabilities such as the ability to outperform the best humans on tasks such as scientific research, business/military/political strategy, engineering, and persuasion/manipulation.

• Planning capabilities such as the ability to make and carry out plans, in pursuit of objectives, as if with an understanding of the world.

• Strategic awareness: the ability to make plans that represent the effect of gaining and maintaining power over humans and the environment.

Carlsmith believes that it is more likely than not that we will be able to build agents with all of the capabilities described above by 2070.3

It could be difficult to align or control these systems

If we create advanced AI systems, how can we ensure they are aligned in the sense that they do what their designers want them to do? Here, misalignment looks less like failing or breaking in an effort to do what the designers want, and more like deliberately doing something designers don't want it to do. It would look less like an employee underperforming, and more like an employee trying to embezzle funds from their employer.

There are several strategies for aligning AI systems, but each of them faces difficulties:

• Aligning objectives: one strategy is to control the objectives that AI systems have. We might give examples of desired behaviour, set up the evolutionary environment for AI carefully, or give feedback on behaviour we like and dislike. These methods, especially when they rely on human feedback, may be difficult to scale to advanced systems. More fundamentally, it may be difficult to avoid situations where the AI system learns to do something that meets our evaluation criteria but with a fundamentally different strategy. For example, we might attempt to train an AI to tell the truth, but accidentally teach it to create fictional internet sources and cater to our biases. The problem is not about ensuring an AI system ‘understands’ what we want, because a sufficiently advanced AI might understand perfectly well and use that knowledge to deceive us in pursuit of its own goals.

• Limiting capabilities: another strategy might be to create agents that have limited capabilities. We are more likely to be able to stop limited AI systems from engaging in misaligned behaviour, and the systems themselves would be less likely to believe that deception would pay off for them.4 The major difficulty here is that we will face strong incentives to create AI systems that are capable and pursue long-term objectives, because these systems will help our scientists, corporations, and politicians use AI to pursue their own goals. In addition, there may also be technical difficulties, and limiting capacities by e.g. ensuring AI systems can only do a narrow range of tasks or only pursue short-term objectives might be more difficult than building systems with advanced capabilities to do similar tasks.

• Controlling incentives: finally, we might try to control the environment in which AI systems are deployed. For instance, we might want to prevent a system from hacking by withholding internet access or turning off any system that is caught hacking. The problem is that controlling incentives becomes more difficult as AI systems become more capable. The example above may prevent a moderately powerful system from hacking, but a sufficiently sophisticated system might realise that there is no realistic chance that we can catch it. 

In addition to the difficulties above, there are several factors that make alignment particularly difficulty compared to other safety problems, such as building nuclear reactors:

• Active deception: AI systems may be trying to deliberately undermine our efforts to monitor and align their behaviour. For instance, a sufficiently advanced system might realise that it needs to pass certain safety tests in order to be released.

• Unpredictability: AI systems may consider strategies that we can't imagine, and they might have cognitive abilities that are opaque to us.

• No room for error: finally, we tend to solve safety problems through trial and error. However, the stakes for aligning advanced AI systems might be extremely high, and therefore we may not be able to learn from mistakes.

Powerful, misaligned AI systems could disempower humanity

Sufficiently advanced AI systems that are misaligned are likely to realise that taking power over humans and the environment will allow them to pursue their other goals, regardless of what those other goals are.5

Power-seeking behaviour might include things like manipulating human attempts to monitor, retrain, or shut off misaligned systems; blackmailing, bribing or manipulating humans; attempting to accumulate money and computational resources; making unauthorised  backup copies of themselves; manipulating or weakening human institutions and politics; taking control of automated factories and critical infrastructure.

Power-seeking, unlike other forms of misalignment, is crucially important because the scale of potential failures is enormous. An AI system that is attempting to take control could, well, take control. Our relationship to these powerful AI systems might be similar to the relationship that chimpanzees have to us: the fate of our friends, family, and community in the hands of a more capable and intelligent species that may not share our values. This would be a very bad outcome for us, perhaps as bad as extinction.6

Knowing this, we still might deploy advanced, misaligned AI systems

It should be reasonably clear that there are strong reasons to avoid deploying advanced misaligned AI systems. There are several reasons to be concerned that they may be deployed anyway:

1. Unilateralist's curse: we might expect that, once some people can build advanced AI systems, the number of people with potential access to those systems will grow over time. Different actors may have different views about how dangerous AI systems are, and the most optimistic and least cautious might end up deploying even if doing so presents clear dangers.

2. Externalities: even if it is in humanities interests to avoid deploying potentially misaligned systems, some individuals might stand to personally gain a lot of money, power, or prestige, and face only a fraction of the cost if things go poorly. This might be similar to how while it would be in humanities interests to reduce carbon emissions, many corporations are incentivised to continue to emit large amounts.

3. Race dynamics: if several groups are competing to build AI systems first, then they might know that they could gain an advantage by cutting corners on expensive or difficult alignment strategies. This could generate a race to the bottom where the first AI systems to be deployed are the quickest and cheapest (and least safe) to develop .

4. Apparent safety: advanced AI systems might offer opportunities to solve major problems, generate wealth, and rapidly advance science and technology. They may also actively deceive us about their level of alignment. Without clear signs of misalignment, it might be difficult to justify ignoring the promise of these systems even if we think they could be manipulating us (in ways that we can't detect). We might also overestimate our ability to control advanced systems.

Of course, if we notice an AI is actively deceiving us and seeking power, we would try to stop it. By the time we deploy advanced AI systems of the kind that pose a significant risk, we are likely to have more advanced tools for detecting, constraining, responding to and defending against misaligned behaviour.

Even so we may fail to contain the damage. First, as AI capabilities increase, we will be at an increasing disadvantage, especially if this happens in hours or days rather than months or years. Second, AI systems may deliberately hide their misalignment and interfere with our attempts to monitor and correct them, and so we may not detect misaligned behaviour early on. Third, even if we do get warning shots, we may fail to respond quickly and decisively, or face problems that are too difficult for us to solve.  Unfortunately, many potential solutions may only superficially solve the problem, by essentially teaching the system to more carefully avoid detection. Finally, all of the factors that lead misaligned systems to be deployed in the first place would contribute to the difficulty of correcting alignment failures after deployment.

 Conclusion

Carlsmith illustrates how AI could lead to human disempowerment:

1.  It could become possible and feasible to build relevantly powerful, agentic AI systems, and we might have strong incentives to do so.

2. It might be much harder to build these systems such that they are aligned to our values, compared to building systems that are misaligned but are still superficially attractive to deploy.

3. If deployed, misaligned systems might seek power over humans in high-impact ways, perhaps to the point of completely disempowering humanity.

Overall, Carlsmith thinks there is a greater than 10% chance that the three events above all occur by 2070. If Carlsmith is right, then we face a substantial existential risk from AI systems within our lifetimes, or our children's lifetimes.

Sources

Nick Bostrom (2012). The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents. Minds & Machines 22.

Ajeya Cotra (2020). Draft report on AI timelines. AI Alignment Forum.

Katja Grace, John Salvatier, Allan Dafoe, Baobao Zhang, & Owain Evans  (2018). Viewpoint: When Will AI Exceed Human Performance? Evidence from AI Experts. Journal of Artificial Intelligence Research 62.

Toby Ord (2020). The Precipice: Existential Risk and the Future of Humanity. Bloomsbury Publishing. See summary here.

Cover image by Pixabay.

Conflict of interest

I have received grants from Open Philanthropy (Carlsmith’s employer), including for work on this blog. Although I asked Carlsmith directly for feedback on this piece,7 Open Philanthropy had no direct input.

The value of the future may be vast. Human extinction, which would destroy that potential, would be extremely bad. Some argue that making such a catastrophe just a little less likely would be by far the best use of our limited resources — much more important than, for example, tackling poverty, inequality, global health or racial injustice.1 In “High risk, low reward: A challenge to the astronomical value of existential risk mitigation”, David Thorstad argues against this conclusion. Suppose the risks really are severe: existential risk reduction is important, but not overwhelmingly important. In fact, Thorstad finds that the case for reducing existential risk is stronger when the risk is lower.

The simple model

The paper begins by describing a model of the expected value of existential risk reduction, originally developed by Ord (2020;ms) and Adamczewski (ms). This model discounts the value of each century by the chance that an extinction event would have already occurred, and gives a value to actions that can reduce the risk of extinction in that century. According to this model, reducing the risk of extinction this century is not overwhelmingly important — in fact, completely eliminating the risk we face this century could at most be as valuable as we expect this century to be.

This result — that reducing existential risk is not overwhelmingly valuable — can be explained in an intuitive way. If the risk is high, the future of humanity is likely to be short, so the increases in overall value from halving the risk this century are not enormous. If the risk is low, halving the risk would result in a relatively small absolute reduction of risk, which is also not overwhelmingly valuable. Either way, saving the world will not be our only priority.

Modifying the simple model

This model is overly simplified. Thorstad modifies the simple model in three different ways to see how robust this result is: by assuming we have enduring effects on the risk, by assuming the risk of extinction is high, and by assuming that each century is more valuable than the previous. None of these modifications are strong enough to uphold the idea that existential risk reduction is by far the best use of our resources. A much more powerful assumption is needed (one that combines all of these weaker assumptions). Thorstad argues that there is limited evidence for this stronger assumption.

Enduring effects

If we could permanently eliminate all threats to humanity, the model says this would be more valuable than anything else we could do — no matter how small the risk or how dismal each century is (as long as each is still of positive value). However, it seems very unlikely that any action we could take today could reduce the risk to an extremely low level for millions of years — let alone permanently eliminate all risk.

Higher risk

On the simple model, halving the risk from 20% to 10% is exactly as valuable as halving it from 2% to 1%. Existential risk mitigation is no more valuable when the risks are higher. 

Indeed, the fact that higher existential risk implies a higher discounting of the future indicates a surprising result: the case for existential risk mitigation is strongest when the risk is low. Suppose that each century is more valuable than the last and therefore that most of the value of the world is in the future. Then high existential risk makes mitigation less promising, because future value is discounted more aggressively. On the other hand, if we can permanently reduce existential risk, then reducing risk to some particular level is approximately as valuable regardless of how high the risk was to begin with. This implies that if risks are currently high then much larger reduction efforts would be required to achieve the same value.

Value increases

If all goes well, there might be more of everything we find valuable in the future, making each century more valuable than the previous and increasing the value of reducing existential risk. On the other hand, however, high existential risk discounts the value of future centuries more aggressively. This leads to a race between the mounting accumulated risk and the growing improvements. The final expected value calculation depends on how quickly the world improves relative to the rate of extinction risk. Given current estimates of existential risk, the value of preventing existential risk receives only a modest increase.2 However, if value grows quickly and we can eliminate most risks, then reducing existential risk would be overwhelmingly valuable — we will explore this in the next section.

The time of perils

So far none of the extensions to the simple model imply reducing existential risk is overwhelmingly valuable. Instead, a stronger assumption is required. Combining elements of all of the extensions to the simple model so far, we could suppose that we are in a short period — less than 50 centuries — of elevated risk followed by extremely low ongoing risk — less than 1% per century, and that each century is more valuable than the previous. This is known as the time of perils hypothesis. Thorstad explores three arguments for this hypothesis but ultimately finds them unconvincing.

Argument 1: humanity’s growing wisdom

One argument is that humanity's power is growing faster than its wisdom, and when wisdom catches up, existential risk will be extremely low. Though this argument is suggested by Bostrom (2014, p. 248), Ord (2020, p. 45) and Sagan (1997, p. 185), it has never been made in a precise way. Thorstad considers two ways of making this argument precise, but doesn’t find that they provide a compelling case for the time of perils.3

Argument 2: existential risk is a Kuznets curve

Figure 1: the existential risk Kuznets curve, reprinted from Aschenbrenner (2020).

Aschenbrenner (2020) presents a model in which societies initially accept an increased risk of extinction in order to grow the economy more rapidly. However, when societies become richer, they are willing to spend more on reducing these risks. If so, existential risk would behave like a Kuznets curve — first increasing and then decreasing (see Figure 1).

Thorstad thinks this is the best argument for the time of perils hypothesis. However, the model assumes that consumption drives existential risk, while in practice technology growth plausibly drives the most concerning risks.4 Without this link to consumption, the model gives no strong reason to think that these risks will be reduced in the future. The model also assumes that increasing the amount of labour spent on reducing existential risks will be enough to curtail these risks — which is at best unclear.5 Finally, perhaps even if the model is correct about optimal behaviour, real world behaviour may fail to be optimal.

Argument 3: planetary diversification

Perhaps this period of increased risk holds only while we live on a single planet, but later, we might settle the stars and humanity will be at much lower risk. While planetary diversification reduces some risks, it is unlikely to help us against the most concerning risks, such as bioterrorism and misaligned artificial intelligence.6 Ultimately, planetary diversification does not present a strong case for the time of perils. 

Conclusion

Thorstad concludes that it seems unlikely that we live in the time of perils. This implies that reducing existential risk is probably not overwhelmingly valuable and that the case for reducing existential risk is strongest when the risk is low. He acknowledges that existential risk may be valuable to work on, but only as one of several competing global priorities.7 

References

Thomas Adamczewski (ms). The expected value of the long-term future. Unpublished manuscript. 

Leopold Aschenbrenner (2020). Existential risk and growth. Global Priorities Institute Working Paper No. 6-2020

Nick Bostrom (2013). Existential risk prevention as a global priority. Global Policy 4.

Nick Bostrom (2014). Superintelligence. Oxford University Press.

Carl Sagan (1997). Pale Blue Dot: A Vision of the Human Future in Space. Balentine Books.

Anders Sandberg and Nick Bostrom (2008). Global Catastrophic Risks Survey. Future of Humanity Institute Technical Report #2008-1

Toby Ord (2020). The Precipice: Existential Risk and the Future of Humanity. Bloomsbury Publishing.

Toby Ord (ms). Modelling the value of existential risk reduction. Unpublished manuscript.

We live at a time of rapid technological development, and how we handle the most powerful emerging technologies could determine the fate of humanity. Indeed, our ability to prevent such technologies from causing extinction could put us amongst the most important people in history. In “Doomsday rings twice”, Andreas Mogensen illustrates how our potential importance could be evidence for humanity’s near-term extinction. This evidence indicates that either extinction would be very likely or that we cannot make a difference to extinction risk.

Responding to evidence

Consider how we respond to new evidence. When you see two people holding hands, you would consider them more likely to be in a relationship than you did previously. Although people sometimes hold hands when they are not in a relationship, you are more likely to observe hand-holding if they are. In general, you should update towards hypotheses that make your observations more likely. Equivalently, seeing two random people holding hands is surprising, but seeing a couple hold hands is mundane. We should update towards hypotheses that would be less surprising given our evidence.

Importance of current people as evidence for near-term extinction

Many have argued that we live in a time of perils.1 Humanity has only just come into immense power––we could destroy ourselves with nuclear bombs or bioweapons. The current generation is incredibly important because we are uniquely placed to prevent human extinction. And it turns out our importance would be evidence for extinction over survival. To see why, consider how surprising our importance would be if we survived. Humanity has enormous potential––the future could contain trillions of people if we survive. If we were amongst the most important of these trillions of potential people, that would be incredibly surprising. It would be less surprising if we were the most important members of a smaller group––and we would be if humanity went extinct in the near term. Evidence favours the hypothesis which would make our observations less surprising, so our importance is evidence for near-term extinction.

Mogensen’s argument is closely related to the ‘original’ doomsday argument, which uses the fact that we are early rather than the fact that we seem unusually influential––see Leslie (1998) and Carter and McCrea (1983). However, Mogensen considers the original argument to have a flaw that his own version does not inherit. The problem with the argument is that the evidence it gives us is usually balanced out by another piece of evidence. The “surprisingly early” fact is entailed by the principle that you should reason as if you are randomly selected from observers in your reference class (see Bostrom, 2002). When you observe that you are surprisingly early, it is particularly surprising if humanity survives for eons. However, this principle is always exactly counterbalanced by another principle, that you ought to assign initially higher credence to worlds with many observers. The intuition is that we would be incredibly lucky to be amongst the only observers in the universe––so you must assign higher probability to the hypothesis according to which your reference class is bigger (see Dieks, 1992). Mogensen notes that another possible objection is that the argument depends on a particular choice of reference class, but he thinks that this is a more general problem rather than a specific objection to this argument (see Goodman, 1983 and Hajek, 2007). However, our apparent importance is an additional piece of evidence that really can change the chance of extinction.

How strong is this evidence?

Mogensen argues the evidence is strong enough to command a dramatic shift in our beliefs. No matter what you thought before, our importance should be evidence that extinction looms larger than previously suspected. We don’t have enough information to know exactly how strong this evidence is, but we can approximate the strength of the evidence through some reasonable estimates. 

From the dawn of time till the end of the world, how many people will there be? There have been about 60 billion so far.2 For simplicity, suppose that if we go extinct soon, there will be 100 billion in total. If we survive, there might be trillions more––suppose 100 trillion, for concreteness. Finally, suppose that regardless of whether or not we prevent extinction, we are amongst the most important 10 billion people to ever live.3

How much more surprising would our survival be given these estimates? Our position amongst the 10 billion most important people would only be a little surprising amongst 100 billion people––we would occupy the top 10%. However, our position would be extraordinarily surprising if humanity survived. A total of 100 trillion people would place us in the most important 0.01%.4 This means, we are about 1,000 times more likely to be in the most important 10% of people than the most important 0.01%.

This is strong evidence––strong enough that it would warrant the dramatic shift from a 5% chance to a 98% chance of extinction. We can explain this using the concept of the “odds ratio”, which is used in betting and statistics, and gives an easy way to update on new evidence. A 5% chance is equivalent to the odds ratio being 1 to 19. We can think of 1 to 19 odds as saying that there are 20 total possibilities (19+1) of which only one involves extinction (so there is a 1/20 or 5% chance of extinction), and 19 involve us surviving the next few centuries (19/20 chance). Our position amongst the 10 billion most important people would be 1,000 times as surprising if we survived the next few hundred years. This moves the odds ratio from 1 to 19 all the way up to 1,000 to 19. That is, of 1,019 total possibilities, 1,000 of them involve extinction. In other words, a 98% chance of extinction. The broader point is that––whatever your initial beliefs––our importance provides very strong evidence for extinction. 

Conclusion

Mogensen’s argument presents our position of relative importance as evidence that we are likely to go extinct in the short-term. This could be because the risk of extinction is higher than previously thought, or because we cannot affect the current extinction risk. If the overall risk is high because we cannot affect it, then we are not living at a time of unique risk––and we are not particularly important. Mogensen thinks that you could take the evidence either way––though he slightly favours the conclusion that we are not so important after all.5

Photo by Johannes Plenio

Effective altruists seek to do as much good as possible given limited resources. Often by donating to important causes like global health and poverty, farmed animal welfare, and reducing existential risks. Can we help more by donating now or later? This is the thorny question William MacAskill tackles in the paper “When should an effective altruist donate?”. He explores several considerations and presents a simple framework to help individual donors and philanthropic organisations decide for themselves. In MacAskill’s view, the most important considerations are:

• Do we have particularly good opportunities to invest in movement-growth?

• How quickly are the best opportunities being funded?

• How will our ability to identify philanthropic opportunities change over time?; and

• To what extent will our values improve over time?

The intuition behind waiting to donate

Some may be surprised that, rather than donating as soon as possible, it can be more effective to invest today and thereby donate more tomorrow. To strengthen our intuition for this idea, consider a slightly different context: an altruistic 18-year-old deciding how to maximise the impact they’ll have on the world. Rather than trying to have an impact immediately, it could be better for them to attend university first. In other words, we might encourage them to invest in their skills and networks at present, so that they can have a larger impact further down the line. Likewise, if returns on investment are sufficiently high, a donor might be able to help a greater number of future people by investing now to donate later.

Important considerations about when to donate

MacAskill identifies six important considerations about when to donate. He then also gives a qualitative framework for weighing these six considerations (p. 14-15).

1. Special relationships

Our special relationship to the present generation might encourage us to assign greater weight to their wellbeing and interests. We have special relationships with our family, friends and conationals. Perhaps we also have special relationships with all people alive today, compared to people in the future. Though moral philosophers typically argue that the interests of future people are in some fundamental sense just as important as our own, MacAskill thinks — given our moral uncertainty — we should still give slightly more weight to the interests of people we have special relationships to. 

2. Changing opportunities

All else equal, we should give when we have the best opportunities. Whether our opportunities are getting better or worse depends largely on two opposing effects. Firstly, the world is getting richer. Over time, this means that the most effective interventions will have already received sufficient funding, and the quality of our remaining opportunities will be worse. Secondly, we are also discovering new interventions (e.g., one of the best current ways to combat malaria is through distributing bed-nets, but in the future we might be able to cheaply eradicate malaria using gene drives). It’s difficult to know which of these effects is stronger, but MacAskill tentatively leans towards thinking our opportunities are getting worse over time. 

3. Getting better knowledge

Even if our opportunities are getting worse over time, we may be getting better at recognising which are our best opportunities. In general, our ability to identify important opportunities has been getting better over time. For instance, we have increasingly run randomised controlled trials which indicate which charitable interventions are the most helpful, and this information is increasingly aggregated and analysed by organisations like Givewell, leading to better information about which charities are likely to be the most cost-effective. This consideration tends towards giving later.

4. Value changes

If you decide to give later, your values may change in the meantime — indeed, many people have different values in their sixties when compared to their twenties. These changes might be viewed in a positive way — perhaps they will be the result of deliberation and improved understanding. They might, however, also be driven by less positive factors — for instance, if you became wealthier and at the same time came to believe that wealthy people shouldn’t donate much at all, then this change could be seen as suspiciously self-serving. Overall, MacAskill thinks that “...we should defer to our future selves on our values, if it’s the case that we think that their values are the result of ‘good’ processes, such as careful reflection, discussion with peers, consideration of moral argument, and so on” (p. 12). However, he also argues that not all value changes are good. 

5. Movement growth

The effect of donations on movement growth — for example by donating to an organisation that uses it to fundraise — can be viewed as a kind of investment that might have exceptional and compounding returns. For example, Giving What We Can estimates that each dollar donated to them eventually results in $100 in donations towards their top charities.1

6. Amount of money to donate

This consideration is deceptively simple in theory: all else equal, you should donate whenever you are able to give the most money. In the paper, MacAskill goes into detail on this category, considering taxes, uncertainty and financial investments. Perhaps surprisingly, weakness of will is relevant here. If you are likely to fail to meet your commitment to giving in the future, due to weakness of will, and you can’t mitigate this by setting up a donor advised fund, then perhaps you should donate earlier, even if it’s less effective, rather than risk failing to donate later.

Photo by Danish Ahmad.

Subscribe now

Longtermist altruists – who care about how much impact they have, but not about when that impact occurs1 – have a strong reason to invest resources before using them directly.2 Invested resources could grow much larger and be used to do much more good in the future. For example, a $1 investment that grows 5% per year would become $17,000 in 200 years.3 However, some people argue that we are living in an unusual time, during which our best opportunities to improve the world are much better than they ever will be in the future. If so, perhaps we should spend our resources as soon as possible. 

In “Are we living at the hinge of history?”, William MacAskill investigates whether actions in our current time are likely to be much more influential than other times in the future.  (‘Influential’ here refers specifically to how much good we expect to do via direct monetary expenditure – the consideration most relevant to our altruistic decision to spend now or later.) After making this ‘hinge of history’ claim more precise, MacAskill gives two main arguments against the claim: the base rate and inductive arguments. He then discusses some reasons why our time might be unusual, but ultimately concludes that he does not think that the ‘hinge of history’ claim holds true.

Source: Our World in Data.4

The base rate argument

When we think about the entire future of humanity, we expect there to be a lot of people, and so we should initially be very sceptical that anyone alive today will be amongst the most influential human beings. Indeed, if humanity doesn’t go extinct in the near future, there could be a vast number of future people – settling near just 0.1% of stars in the Milky Way with the same population as Earth would mean there were 10²⁴ (a trillion trillion) people to come. Suppose that, before inspecting further evidence, we believe that we are about as likely as anyone else to be particularly influential. Then, our initial belief that anyone alive today is amongst the million most influential people would be 1 in 10¹⁸ (1 in a million trillion). 

From such a sceptical starting point, we would need extremely strong evidence to become convinced that we are presently in the most influential time era. Even if there were only 10⁸ (one hundred million) people to come, then in order to move from this extremely sceptical position (1 in 10⁸) to a more moderate position (1 in 10), we would need evidence about 3 million times as strong as a randomised control trial with a p-value of 0.05.5  MacAskill thinks that, although we do have some evidence that indicates we may be at the most6 influential time, this evidence is not nearly strong enough.

The inductive argument

There is another strong reason to think our time is not the most influential, MacAskill argues:

Premise 1: Influentialness has been increasing over time.

Premise 2: We should expect this trend to continue.

Conclusion: We should expect the influentialness of people in the future to be greater than our own influentialness.

Premise 1 can be best illustrated with an example: a well educated and wealthy altruist living in Europe in 1600 would not have been in a position to know about the best opportunities to shape the long-run future. In particular, most of the existential risks they faced (e.g. an asteroid collision or supervolcano) were not known, nor would they have been in a good position to do anything about them even if they were known. Even if they had the scientific knowledge that we have, they might have used it to pursue a worse moral view.7 Indeed, it is likely that future generations will discover ways in which we are misguided, both morally and scientifically. If we are mistaken enough, our (well-intentioned) present actions could actually be doing harm. Premise 2 indicates that we should expect the trend of improvement to continue. This is especially plausible because we can identify gaps in our scientific, technological and moral understanding. Overall, this argument indicates that we should expect future generations to be more influential than we are.

Reasons why our time might be unusual

MacAskill also discusses several reasons one might think that our time is unusual, and therefore may be unusually influential. Our time is unusual because we currently live on a single planet, while most people who will ever live will likely (in expectation) be part of an interplanetary civilisation.8 We also live at a time of extreme technological progress which cannot continue indefinitely: our current economic growth rate is around 3.5%, but 2% annual growth over the next 10,000 years would result in an economic output of 10¹⁹ (ten million trillion) times the current world GDP for every atom in the galaxy.

There are three important ways in which this could make our time unusually influential: 

1. Our single planet is a single point of failure, which may make the risk of extinction temporarily higher than usual.9

2. While we live on a single planet, the most influential people today may have an unusual ability to influence humanity as a whole – both because they can communicate near instantaneously with almost everyone and because their resources are a relatively large fraction of the total. If humanity becomes a much larger space-faring civilisation, these will likely both change.  

3. Plausibly, the fate of the future will be decided by how we handle some particular technology (such as artificial intelligence or particularly dangerous new weapons) and we are more likely to discover such a technology during a period of rapid growth.10  

However, each of these arguments has important caveats. In relation to the first argument, most people who are worried about existential risk believe that a large part of the risk comes from misaligned artificial intelligence,11 and this would not be significantly reduced by planetary diversification. In relation to the second argument, this period of unusual influence may be prolonged if our civilisation stays earthbound for thousands of years or it just takes longer than we expect to leave the solar system. (It only takes one hour for light to traverse the full diameter of the asteroid belt, so the ability of the most influential people to influence humanity as a whole may remain high for quite some time.) In relation to the third argument, perhaps this period of remarkable economic growth will last longer than most anticipate. Even if this period is short, one could argue that longtermists will be less influential during periods of high economic growth, because the unpredictability of a rapidly changing environment hinders the execution of very long-term projects. Overall, MacAskill thinks that these arguments provide evidence that our time may be the most influential. However, the base rate and inductive arguments show that we should be extremely sceptical that we live at the most important time – and the evidence presented in this section does not seem strong enough to overcome these arguments.

Overall, we probably do not live at the ‘hinge of history’. If we did, this would give us a powerful reason to spend now rather than investing to have a much larger impact later. Instead, the case for investment remains strong.

References

Daniel Benjamin et al. (2018) Redefine statistical significance. Nature Human Behaviour. 2. 

Nick Bostrom (2014). Superintelligence: Path, Dangers, Strategies. Oxford University Press.

Hilary Greaves and William MacAskill (2021). The case for strong longtermism. GPI Working Paper No. 5-2021.

William MacAskill (2022). Are we living at the hinge of history? Ethics and Existence: The Legacy of Derek Parfit. Oxford University Press. Edited by Jeff McMahan, Tim Campbell, James Goodrich, and Ketan Ramakrishnan.

William MacAskill (2019).When should an effective altruist donate? GPI Working Paper No. 8-2019.

Toby Ord (2020). The Precipice: Existential Risk and the Future of Humanity. Bloomsbury Publishing.

Carl Sagan (1994). Pale Blue Dot: A Vision of the Human Future in Space. Random House.

Philip Trammell (2021). Dynamic public good provision under time preference heterogeneity: theory and applications to philanthropy. GPI Working Paper No. 9-2021.

As longtermists, we believe it is crucial to shape the long-term future for the better. Legal longtermists are particularly interested in how the law can be used to safeguard the interests of future generations and improve the lives of people in the coming centuries and millennia. In “Protecting future generations”, Eric Martínez and Christoph Winter find that legal academics tend to believe that future generations deserve greater legal protections.

Do future generations deserve more protection?

Longtermism is the idea that we should focus on making the future better. This is because the future is incredibly important — and if we can make it better in predictable ways, then we could impact the lives of billions or trillions of people. Legal longtermists focus on how legal systems can protect future generations and improve the long-run future. This survey looks at a few open questions in legal longtermism — for example, can and should the law protect future generations? To answer these questions, Martínez and Winter surveyed 516 legal experts from top English-speaking and common-law universities around the world.1

Legal systems tend not to protect future generations.2 For example, in the United States, cost benefit analysis is used to make regulatory decisions — but the process “discounts” away the interests of future generations3 in a way seems unjustified to most philosophers.4

In the legal system, one of the most important rights is the ability to take legal action. In the United States, this right can be invoked when you are harmed or placed in immediate danger. Harms to future generations don't fit this criteria.5 However, over two-thirds of surveyed legal experts thought there was some legal basis for people in the next 100 years to sue, in at least some cases. Strikingly, more than half also thought the same was true for those living more than 100 years from now. This may indicate opportunities to assert the rights of future generations with the right cases and arguments.6

However, future generations have never been allowed to sue (Bogojević, 2020), and legal experts think that future generations should be protected more than they are today. On average, future generations are only protected a third as much as they should be — according to legal experts. They thought that the current generation should be protected more than they are now, and more than future generations should be. They also found that other groups, like animals and people from other countries, are not protected as much as they should be, but future generations are by far the most neglected by the law.

Protecting future generations

The law sometimes has long-term effects, for example, ancient Roman law still influences many civil law systems (Watson 1991). In other cases, effects may be short or unpredictable.7 For the legal longtermist, everything hinges on whether the law provides predictable and feasible ways to help future generations. Legal experts think the law can provide protection to future generations, and that it is one of the best ways to predictably help future generations. Martínez and Winter estimate8 that around 74% of legal academics somewhat agree that the law can help people more than a century from today. While about 73% somewhat agree that legal methods are among the most predictable and feasible ways of doing so. About 41% somewhat agree there are even ways for the law to protect people who are a millennium away!9

Next, Martínez and Winter explored the most promising areas of law from the perspective of legal longtermism. Many show at least some promise — for each area in the survey at least half of legal academics would somewhat agree that area could be helpful. However, legal academics were most confident in environmental law and constitutional law. The most promising constitutional mechanisms are ensuring that 1% of GDP goes towards safeguarding humanity against existential risks or giving explicit legal standing to future generations.10 However, the differences here were small.

Finally, they asked about causes that longtermists tend to find important, to identify the most promising ones for legal intervention. Although at least half of the academics thought all the areas might be helped, they had lower confidence in artificial intelligence (56%) than the others — for instance, climate change (84%).

Conclusion

Eric Martínez and Christoph Winter surveyed legal academics and found that they generally believe that future generations deserve more protection. The law provides a promising avenue for providing these protections — in particular environmental and constitutional law. Future research could explore the attitudes of different groups towards legal longtermism, try to compare specific policies rather than broad areas of law, or investigate other legal concepts such as personhood.

Cover photo by Colin Lloyd.

Sources

Renan Araújo & Leonie Koessler (2021). The Rise of the Constitutional Protection of Future Generations. Legal Priorities Project Working Paper No. 7-2021.

Sanja Bogojević (2020). Human rights of minors and future generations: global trends and EU law particularities. Review of European comparative & international environmental law 29/2.

John Broome (1994). Discounting the Future. Philosophy & Public Affairs 23/2.

Tyler Cowen & Derek Parfit (1992). Against the Social Discount Rate. Philosophy, Politics, and Society: Volume 6, Justice Between Age Groups and Generations. Yale University Press. Edited by Peter Laslett & James S. Fishkin.

Moritz A. Drupp, Mark C. Freeman, Ben Groom and Frikk Nesje (2018). Discounting Disentangled. American Economic Journal: Economic Policy 10/4.

Thomas Ginsburg, Zachary Elkins, and James Melton (2009). The Lifespan of Written Constitutions. The University of Chicago Law School.

Andreas Mogensen (2019). Maximal Cluelessness. Global Priorities Institute Working Paper No. 2-2019.

Office of Management and Budget (2003). Executive Office of the President, Circular A-4: Regulatory Analysis.

Derek Parfit (1986). Reasons and Persons. Oxford University Press.

Alan Watson (1991). Roman Law and Comparative Law. University of Georgia Press.

A single human in the wilderness is nothing exceptional. But together humans have the ability to shape the world and determine the future of our species, planet, and universe.

We learn from our ancestors, add minor innovations of our own, and teach our children. We are the beneficiaries of countless improvements in technology, mathematics, language, institutions, culture, and art. These improvements make our lives much better than the lives of our ancestors.1

We hope that life will continue to improve. And we could have a lot of time to get things right. Humans have walked the earth for around 200,000 years, but a typical mammalian species lasts for a million years, and our planet will remain habitable for a billion years. This is enough time to eradicate malaria and HIV, eliminate depression and dementia, and create a world free from racism, sexism, torture, and oppression. With so much time ahead of us, we might even figure out how to leave our solar system and settle the stars. If so, we could have a truly staggering number of descendants who can explore the universe and build wonders and masterpieces better than we can imagine. If we go extinct, all of this will be lost.

We have always faced a small risk from asteroids, pandemics, and volcanoes. But it was only recently that we began to face larger risks of our own creation. This period of heightened risk began last century with the invention of nuclear weapons (we now have enough to kill everyone on earth). Over the next century we will face additional risk from emerging developments in biotechnology and AI. In the words of Toby Ord, we are standing on “a crumbling ledge on the brink of a precipice.”

Safeguarding humanity is the defining challenge of our time.2 If we rise to it, there may be trillions of people living meaningful lives in the future. If we fail, then in all likelihood we will destroy ourselves. The fate of the world rests on our collective decisions.

Subscribe now

Why should we try to prevent extinction?

If a large asteroid was hurtling towards Earth, few would argue against building a deflection system. This indicates that our collective inaction is driven by a shared sentiment that the risk of extinction is low, rather than the belief that humanity is not worth protecting. However, it is still worth reflecting on why preventing extinction is so important.

A tragedy on the grandest scale

Sudden extinction, such as from an asteroid collision, would involve the sudden and gruesome deaths of billions of people, perhaps everyone. This alone would make it the most severe tragedy in history.

The destruction of our potential

Extinction would destroy our immense potential. Almost all humans that will ever live are yet to be born. Almost all human well-being and flourishing is yet to happen.3 All of this would be lost if the present generation went extinct.

Intergenerational projects

Our ancestors set in motion great projects for humanity — ending war, forging a just world, and understanding the universe. No single generation can complete these projects. But humanity can, with each generation contributing just a little. We benefit immensely from knowledge and wisdom passed down to us from previous generations, and we owe it to our children and grandchildren to protect this legacy and pass it down to them. Extinction would also destroy all cultural traditions, languages, poetry, and culture. We ought instead to protect, preserve, and cherish these things.4

Civilisational virtues

We are accustomed to understanding virtues on an individual level, but we could also think of the collective virtues of humanity. When we fail to take these risks seriously, humanity might collectively demonstrate a lack of prudence. When we value our own generation so much as to put all future generations at risk, we demonstrate a lack of patience. And when we fail to prioritise well-known risks, we display a lack of self-discipline. When we do not rise to the challenge, we display a lack of hope, perseverance, and responsibility for our own actions.

Cosmic significance

We may be alone in the universe. If there are no aliens, then all life on Earth may have cosmic significance. Humanity would be in a unique position to explore and understand the universe. We would also have a responsibility to all life, as we would be the only ones who could protect it from harm and promote flourishing on other planets.

Uncertainty

Correctly accounting for our uncertainty about the future tends to strengthen the case for protecting our potential because the stakes are asymmetrical: overinvesting in safety is simply much better than letting everyone die. This means that even if we believe the risks are low, but we are not completely confident, then some efforts to safeguard humanity are warranted.5

Why are existential risks neglected?

Are existential risks neglected?

Humanity spends less money attempting to prevent existential risk than it does on ice cream each year.6 The most risky emerging technologies are biotechnology and AI (see parts 2 & 3). Yet the international body responsible for the continued prohibition of bioweapons has an annual budget less than that of an average McDonald’s restaurant.7 And while we spend billions of dollars improving the capabilities of AI systems, we only spend tens of millions of dollars on ensuring safety. Research similarly neglects the most severe risks: for instance, there is plenty of research on the possible effects of climate change, but scenarios involving more than six degrees of warming are rarely studied or given space in policy discussions (King et al., 2015). There are several reasons for this neglect.

Existential risk as a global public good

When one organisation or government reduces the risk, they improve the situation for everyone in the world. Everyone is incentivised to wait for someone else to solve the problem and benefit from the hard work of others. This dynamic happens across generations too. So, many of the people who benefit if we safeguard humanity have not even been born yet. We do not yet have robust ways to coordinate on these issues.

Short-term institutions

Additionally, political decisions are notoriously short term. Existential risk tends to be ignored in favour of more urgent issues. That said, most existential risks are new relative to our political institutions, which have been built up over thousands of years. We only began to have the power to destroy ourselves in the midde of the last century, and since then there has begun to be serious thought about the possibility of extinction. Perhaps our institutions and practices will gradually adapt.

Patterns of thinking

Our brains are not built to grasp these risks intuitively, and there are several patterns of thinking that lead us to neglect existential risk. For instance, we tend to estimate the likelihood of an event based on how easy it is to recall examples of it happening in the past. This availability heuristic serves us well most of the time, but when we are dealing with risks like extinction, these heuristics allow us to ignore even large and growing risks. We also lack sensitivity to the scale of various catastrophes.

Sources

Biological Weapons Convention Implementation Support Unit (2019). Biological Weapons Convention—Budgetary and Financial Matters.

Mark Nathan Cohen (1989). Health and the Rise of Civilization. Yale University Press.

Joe Hasell, Max Roser, Esteban Ortiz-Ospina and Pablo Arriagada (2022). Poverty. Our World in Data. (This article has been updated since The precipice was published in 2020).

David King, Daniel Schrag, Zhou Dadi, Qi Ye and Arunabha Ghosh (2015). Climate Change: A Risk Assessment. Centre for Science and Policy.

IMARC Group (2019). Ice Cream Market: Global Industry Trends, Share, Size, Growth, Opportunity and Forecast 2019–2024.

McDonald’s Corporation (2018). Form 10-K. (McDonald’s Corporation Annual Report).

Max Roser and Esteban Ortiz-Ospina (2019). Literacy. Our World in Data.

World Health Organization (2016). World Health Statistics 2016: Monitoring Health for the SDGs, Sustainable Development Goals.

Notes

Image of the earth from: www.tobyord.com/earth

Subscribe now

To communicate his impression of the risks accurately, Ord puts numbers on them. These numbers represent his best guesses about the order of magnitude of each risk, based on the research behind his book. They do not represent highly certain estimates of the risks, and new information could easily change them.1

These estimates indicate risks from anthropogenic sources (such as nuclear war) tend to be much more dangerous than risks from natural sources (such as asteroids). In fact, nuclear war, climate change, and environmental damage are each at least as dangerous as all natural risks combined. And together, they pose a risk 1,000 times greater than all natural sources combined.

Even within anthropogenic risks, some technologies pose greater risks than others. Engineered pandemics, unaligned artificial intelligence, as well as uncategorised and unforeseen anthropogenic risks each present around 100 times the risk from any other source.2

Ord’s estimate of the total extinction risk is one in six this century. This may sound pessimistic, but it implies that we have a five in six chance of surviving this century.

It is like we are playing Russian roulette with enormous stakes.

Ord is optimistic, and his estimate assumes that we will recognise the importance of reducing existential risks and take significant steps to reduce them. If we shut our eyes and maintain business as usual, Ord believes we face risks about twice as high — like playing Russian roulette with two bullets in the chamber. But if we get our act together, we could remove both bullets and safeguard humanity.

The next post in this series will explore why existential risks are so important to prevent, our place in the story of humanity, and why existential risks remain neglected today.

Image of the earth from: www.tobyord.com/earth

We saw in the last post that our 2,000-century track record allows us to estimate that the risk of extinction from natural disasters must be very low. What about our track-record of anthropogenic (human-caused) risks? It has been less than 3 centuries since the Industrial Revolution and less than a century since the invention of nuclear weapons, so our track record is compatible with a 50% risk per century. Instead of looking at this track record, we need to look at the details of these risks.

Nuclear weapons

At 3 a.m. one morning in 1979, four independent US command centres saw many incoming nuclear warheads. They only had minutes to respond before the bulk of their own missiles would be destroyed by the incoming strike. When they checked the raw data from the early-warning systems, they realised that there was no attack, and a realistic simulation of a Soviet attack had accidentally been streamed to their system (Brezhnev, 1979; Gates, 2011; Schlosser, 2013).

Cold War tensions led us astonishingly close to nuclear war over 32 times (US Department of Defense, 1981). What would happen if a nuclear war occurred? The worst-case scenario is an all-out nuclear exchange between two countries with many nuclear weapons, such as the US and Russia. This would kill tens or even hundreds of millions of people in the cities hit by the bombs. Radioactive dust would be blown outward, spreading deadly radiation. Smoke from the burning cities would darken the skies, block out the sun, and cool the earth.

This would be unlikely to result in extinction. Most people living outside of major cities in the countries that were bombed would survive the initial blast; the blasts wouldn’t produce enough radioactive dust to make the entire earth inhospitable. The worst effects would be from the darkening of the sky and the ensuing nuclear winter. Our best models suggest that the growing season might be too short for most crops, in most places, for five years (Robock, Oman & Stenchikov, 2007). Billions of people would be at risk of starvation, but humanity would likely survive by growing less efficient crops, building greenhouses, fishing, and perhaps even farming algae.1

Climate change

Carbon dioxide, together with water vapour and methane, creates a kind of gaseous blanket around Earth. This is essential for life (without it, Earth would be a frozen wasteland). Since the Industrial Revolution, we have burned fossil fuels and rapidly increased the amount of carbon dioxide in the atmosphere from about 280 parts per million to 412 parts per million in 2019 (Lindsey, 2018; National Oceanic and Atmospheric Administration, 2019). Unless we significantly reduce emissions, this will quickly warm the planet. While this probably won’t be the end of humanity, there is substantial uncertainty about how much we might emit and what effect it will have.

The Intergovernmental Panel on Climate Change (2014) estimates that a fourfold increase from preindustrial carbon dioxide levels has a two-thirds chance of warming Earth between 1.5 and 9 degrees Celsius (and therefore a one-third chance of warming above 9 degrees or below 1.5 degrees). This is before considering feedback loops, such as increased bushfires, which release additional carbon, and the melting of ice which contains trapped greenhouse gases. This leaves us with a substantial chance of very significant warming. Even though it would be an unprecedented disaster, and this is reason enough to stop emissions, even 20 degrees of warming would leave many coastal areas habitable all year round.

One particularly bad feedback loop involves increased temperatures evaporating water in the oceans, creating a denser blanket of water vapour around Earth and accelerating warming.2 Though current research suggests such an effect would not be strong enough to entirely evaporate the ocean, and probably won’t happen at all, we cannot rule it out. While this is the only known mechanism for climate change to cause the extinction of humanity, there may be unknown mechanisms, and given that Earth has never seen such a rapid period of warming, we have substantial uncertainty about the eventual effects.

Environmental damage

The world’s population grew ever faster between 1800 and 1968. Seeing this, Paul Ehrlich predicted that in the coming decades this growing population would become unsustainable and there would be “an utter breakdown of the capacity of the planet to support humanity.”3

Because of improvements in agriculture and slower population growth, this breakdown has not yet happened, and the global population is expected to peak at 11 billion. This is still many more people than Earth has ever supported, and as people grow in material wealth, the per-person strain on the environment increases beyond what it ever was. Does this present an existential risk?

Resource depletion is unlikely to present any real risk to our potential.4 A bigger threat is biodiversity loss. Some suggest we are witnessing the next mass extinction. While it is difficult to compare current extinction rates with the fossil record, current species loss appears to be both much smaller (1% of species versus at least 75%) and much faster (10 to 100 times faster) than previous mass extinctions. From the perspective of humanity’s survival, the most important thing is that ecosystems do not break down so far that they stop providing vital services such as purifying water, providing energy and resources, improving our soil, and creating breathable air. These risks are not well understood, and if we continue to put enormous pressure on our environment this century, this may result in large, currently unforeseen risks.

Emerging pandemic risk

From 1347 to 1353, between one-quarter and one-half of Europeans were killed by plague (Ziegler, 1969). After World War I, the 1918 flu (also known as Spanish flu) spread to six continents, infected over a third of the world’s population, and killed more people than the war (Taubenberger & Morens, 2006). Neither of these events was devastating enough to end humanity, or even collapse civilisation, and we would likely recover from a pandemic on a similar scale. We can also infer from the fossil record that, like other natural risks, the risk from a natural pandemic must be incredibly low.5 However, improvements in biotechnology mean that we face a substantial new risk from engineered pandemics.

One risk comes from well-intentioned scientists trying to study viruses. Although most of this research poses no danger to humanity, a few experiments involve trying to give viruses new abilities — for instance, making them more deadly or transmissible.6 While these experiments take place in the highest-security labs, there have been multiple leaks of deadly pathogens such as smallpox (1971 and 1978) and anthrax (1979 and 2015). This is particularly worrying because these labs lack transparency and we very likely do not know about all of the leaks.

There is also the threat of misuse. Fifteen countries are known to have developed bioweapons programs at some point in the last century. The largest program was in the Soviet Union, with a dozen labs employing 9,000 scientists to weaponise diseases like plague and smallpox.7 That we have seen few deaths from bioweapons so far, compared to natural pandemics, is not as reassuring as it first appears. Deaths from war follow a power-law distribution where most deaths occur in just a few very large wars and most wars kill far fewer people; if this is the case with biological risks, then the risk might be quite high despite few deaths so far.

Biotechnology is also increasingly democratised. It took 13 years from — 1990 to 2003 — and over $500 million to produce the full DNA sequence of the human genome. In 2019 it cost less than $1,000 and took less than an hour. While this trend will result in fantastic applications of this technology to improve our lives, over the coming century it will also give people increasing access to dangerous pathogens.

There are clear efforts to reduce these risks, but more is needed. For instance, the Biological Weapons Convention of 1972 is monitored by just four employees with a budget smaller than an average McDonald’s restaurant.8 And while many companies that synthesize DNA are careful to ensure that pathogens don’t fall into the wrong hands, perhaps only 80% of their orders are screened for dangerous pathogens (DiEuliis, Carter & Gronvall, 2017).

Artificial intelligence

Initially, AI dominated tasks that were previously thought to require our unique human intelligence, such as chess. But progress was faltering and slow on seemingly simple tasks, such as recognising a dog versus a cat. Now, AI can do many of these tasks — for instance, recognising faces better than a human.9 Experts even find it plausible that we could invent a fully general artificial intelligence this century; a 2016 survey of researchers who had published in NeurIPS and ICML indicated that the average respondent gave this a 50% chance by 2061 (Grace et al., 2018).

Chimpanzees are not going to decide the fate of the world, or even the fate of chimps. Instead, we get to decide, because we are the most intelligent and technologically advanced species. In the absence of other evidence, we should expect that losing our position as the most intelligent species would be a big deal (and perhaps not one that will favour humans).

Importantly, ensuring that AI is aligned to human values appears to be a difficult and unsolved problem. The methods we have for producing intelligence tend to either involve letting a human decide on a reward function and then training a neural network to act in ways that produce greater rewards or letting an AI observe human choice and infer a reward system based on this. But humanity’s values are too complex and subtle to write down as a simple formula, and we do not know how to guide an AI system to learn them.10

Initially, we may have the option to turn off an AI system. But over time, these systems will likely become resistant to this. Indeed, to maximise their reward function they must survive and thwart our attempts to bring their reward function in line with human values. Ultimately, an AI system is incentivised to take control of resources and shape the world — wresting control from humans. Since humans would predictably interfere with these goals, it would also be incentivised to hide its true goals until they are powerful enough to resist our attempts to stop them. 

Contrary to Hollywood blockbusters, AI would not need robots in order to gain control. The most powerful figures of history were not the strongest; Hitler, Stalin, and Genghis Khan used words to convince millions to fight their battles. AI could well do the same. Even if many humans are left alive, this could permanently destroy humanity’s potential — and thus be an existential catastrophe.

Dystopian scenarios

We could lose humanity’s potential by letting the world become locked into a permanent state of little value. The most obvious scenario is permanent authoritarian rule made possible by advances in technology for detecting and eliminating dissent.11 Even if there was a slim chance that we were able to recover, an event like this could destroy most of our potential. There is little such risk in the near future, but as technology improves, this may change.

The next post in this series will tie everything together with an overview of the risk landscape, including quantitative estimates of the risk from nuclear war, climate change, pandemics, and artificial intelligence. 

Sources

Biological Weapons Convention Implementation Support Unit (2019). Biological Weapons Convention—Budgetary and Financial Matters.

Diane DiEuliis, Sarah R. Carter, and Gigi Kwik Gronvall (2017). Options for Synthetic DNA Order Screening, Revisited. mSphere 2/4.

Leonid Brezhnev (1979). Brezhnev Message to President on Nuclear False Alarm, Diplomatic Cable (No. 1979STATE295771) from Sec State (D.C.) to Moscow American Embassy. National Security Archive, United States Department of State.

Matthew Collins, Reto Knutti, Julie Arblaster, Jean-Louis Dufresne, Thierry Fichefet, Pierre Friedlingstein, Xuejie Gao, William J Gutowski Jr., Tim Johns, Gerhard Krinner, Mxolisi Shongwe, Claudia Tebaldi, Andrew J Weaver and Michael Wehner (2013). Long-Term Climate Change: Projections, Commitments and Irreversibility. In Climate Change 2013—The Physical Science Basis: Working Group I Contribution to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press.

Robert M Gates (2011). From the Shadows: The Ultimate Insider’s Story of Five Presidents and How They Won the Cold War. Simon and Schuster.

Katja Grace, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans (2018). Viewpoint: When Will AI Exceed Human Performance? Evidence from AI Experts. Journal of Artificial Intelligence Research 62.

Sander Herfst, Eefje J A Schrauwen, Martin Linster, Salin Chutinimitkul, Emmie de Wit, Vincent J Munster, Erin M Sorrell, Theo M Bestebroer, David F Burke, Derek J Smith, Guus F Rimmelzwaan, Albert D M E Osterhaus and Ron A M Fouchier (2012). Airborne transmission of Influenza A/H5N1 Virus Between Ferrets. Science 336/6088.

Intergovernmental Panel on Climate Change (2014). Summary for Policymakers. In Climate Change 2014—Impacts, Adaptation and Vulnerability: Part A: Global and Sectoral Aspects: Working Group II Contribution to the IPCC Fifth Assessment Report. Cambridge University Press.

Rebecca Lindsey (2018). Climate Change: Atmospheric Carbon Dioxide. Climate.gov.

Charles C Mann (2018). The Book that Incited a Worldwide Fear of Overpopulation. Smithsonian Magazine.

McDonald’s Corporation (2018). Form 10-K. (McDonald’s Corporation Annual Report).

National Oceanic and Atmospheric Administration (2019). Global Monthly Mean CO2. Global Monitoring Laboratory.

Max Popp, Hauke Schmidt and Jochem Marotzke (2016). Transition to a Moist

Greenhouse with CO₂ and Solar Forcing. Nature Communications 7.

Alan Robock, Luke Oman and Georgiy L Stenchikov (2007). Nuclear Winter Revisited with a Modern Climate Model and Current Nuclear Arsenals: Still Catastrophic Consequences. Journal of Geophysical Research: Atmospheres 112/D13.

Eric Schlosser (2013). Command and Control: Nuclear Weapons, the Damascus Accident, and the Illusion of Safety. Penguin.

Jeffery K Taubenberger and David M Morens (2006). 1918 Influenza: The Mother of all Pandemics. Emerging Infectious Diseases 12/1.

US Department of Defense (1981). Narrative Summaries of Accidents Involving US Nuclear Weapons (1950–1980). Homeland Security Digital Library.

Philip Ziegler (1969). The Black Death. Harper Collins.

Image of the earth from: www.tobyord.com/earth

Notes

What is an existential risk?

An existential catastrophe is any event that would destroy humanity’s potential. This could take a few forms:1

• Ordinary extinction: Every human on Earth dies, or there are too few survivors to repopulate it.

• Permanent civilisational collapse: An enormous catastrophe collapses civilisation and severely damages the environment in a way that makes it impossible to rebuild. This would be a world without writing, cities, and law. A collapse of civilisation might or might not be an existential catastrophe; it depends on whether we can rebuild.

• A world in chains: The entire world is locked under totalitarian rule. Advanced technology allows permanent and powerful indoctrination, surveillance, and enforcement, leaving no chance for an uprising and no internal or external pressure to change. Like civilisational collapse, this presents an existential catastrophe if the situation is permanent. 

How can we estimate the danger?

One way is to assume the risk is negligible until there is strong scientific evidence determining that it is higher. This ensures that risks are not exaggerated but does not usually reflect our current understanding of the risks and might lead to dangerous underestimation of emerging risks.2 Instead, Toby Ord begins with an initial impression of the size of the risk, then adjusts this estimate according to the scientific evidence.

The sources of natural extinction risk

Asteroids and comets

Sixty-six million years ago an asteroid hit Earth off the coast of Mexico, burning everything within 1,000 kilometres. The worst effects were caused by a billowing cloud of dust and ash (and sulphate aerosols from the vaporised sea floor) which blocked out the sun and cooled Earth. In the end, every land vertebrate over five kilograms went extinct (Longrich, Scriberas and Wills, 2016).

Supervolcanic eruptions

The very largest volcanic eruptions don’t look like typical volcanoes. Instead of mountains spilling out molten rock, supervolcanoes collapse into a vast craterlike depression (a well-known example is the Yellowstone caldera). One of these eruptions happened 74,000 years ago in Indonesia. Glowing rocks rained down as far as 100 kilometres away, and places as far away as India were covered in a metre-thick blanket of ash. Although this was not close to being an extinction-level event, supervolcanic eruptions present a small risk of civilisational collapse.3 Even though we could likely rebuild civilisation, most of the extinction risk here is driven by the possibility of permanent civilisational collapse.  

Stellar explosions

Sometimes large stars explode, instantly releasing the same amount of energy as our sun will over its 10-billion-year lifetime. If this happened close to Earth, it could alter the climate and erode the ozone layer, leaving us exposed to UV radiation.

Subscribe now

Estimating natural extinction risk

There are many other potential dangers.4 And our understanding of natural risks is recent and growing. It was only in the 1960s that we learned that Earth may have been hit by a large asteroid and we detected the first signs of the bursts of energy emitted by exploding stars. There has been no slowdown in our discovery of new risks, and we do not know what caused several historical mass-extinction events. We should expect to learn about new sources of extinction risk in the coming decades.

Luckily, we can estimate the total natural extinction risk without complete knowledge of the individual risks by examining our track record. Homo sapiens have survived for over 200,000 years. If the risk had been 1% per century, then there would have been a 99.9999998% chance that we would have gone extinct by now. Based on this, we can be extremely confident that the risk is below 0.34% per century, and our best guess is that the risk is below 0.05% per century.5

We might also consider that humans have spread to diverse environments all over the planet, so it’s likely that only mass-extinction events truly threaten us. There have been five of these events since complex life developed — over 540 million years ago— making the extinction risk one in a million (0.0001%) per century.

Where possible, we can supplement this track record with our scientific understanding of the risks to get estimates for individual risks that are sometimes substantially lower than our track record suggests. For instance, we have identified about 95% of the asteroids less than 10 kilometres in diameter and likely all asteroids greater than 10 kilometres across; and we know none are going to hit us this century. Astronomers have also estimated the chances of a stellar explosion close enough to destroy 30% of the ozone layer at about one in 5 million.6

Overall, the picture is incredibly reassuring. While it would be prudent to continue to improve our scientific understanding of these risks and monitor them, these risks are very small over the next century.

The next post will begin to explore the existential risks caused by nuclear weapons, climate change, advanced biotechnology, and artificial intelligence.

Sources

Nick Bostrom (2002). Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards. Journal of Evolution and Technology 9.

Nick Bostrom (2013). Existential Risk Prevention as Global Priority. Global Policy 4/1.

Dario Buttazzo, Giuseppe Degrassi, Pier Paolo Giardino, Gian F Giudice, Filippo Sala, Alberto Salvio & Alessandro Strumia (2013). Investigating the Near-Criticality of the Higgs Boson. Journal of High Energy Physics 2013/89.

Nicholas R Longrich, J Scriberas, and Matthew A Wills (2016). Severe Extinction and Rapid Recovery of Mammals across the Cretaceous-Palaeogene Boundary, and the Effects of Rarity on Patterns of Extinction and Recovery. Journal of Evolutionary Biology 29.

Max Tegmark and Nick Bostrom (2005). Is a Doomsday Catastrophe Likely? Nature 438.

Image of the earth from: www.tobyord.com/earth

Notes

In a sentence: this blog delivers simple explanations of global priorities research.

Subscribe now

What kind of research will the blog cover?

This blog will cover topics that are important for understanding our values and making the world a better place. The focus will be more academic than practical. For example, I might cover research that estimates the probability that we live in a simulation—this is highly speculative, but perhaps has important implications.

As the name suggests, this blog will likely skew towards longtermism—the view that one of our priorities should be helping future generations to survive and flourish. For example, I might cover research into whether future people matter, predicting and impacting the future, and how important we are relative to other people throughout history.

Who should read it?

This blog is most focused on helping motivated students to understand key research about how to do good effectively, although it should also be helpful to anyone interested in global priorities research.

Who writes it?

Riley Harris is a PhD student in Philosophy at Oxford University, and has worked on research communications projects the Global Priorities Institute, Legal Priorities Project, and Longview Philanthropy.

Where should I begin?

Wherever you want really, perhaps the series on Toby Ord’s “The precipice” is a good place to begin.

I want to help… how can I help?

This is a tiny project, as of September 2023 I have around 60 subscribers. I would love for you to spread the good word. If you know someone who would find it valuable, I’d really appreciate it if you sent them a link!

Share