A model of pathways to artificial superintelligence catastrophe for risk and decision analysis

Abstract

An artificial superintelligence (ASI) is an artificial intelligence that is significantly more intelligent than humans in all respects. Whilst ASI does not currently exist, some scholars propose that it could be created sometime in the future, and furthermore that its creation could cause a severe global catastrophe, possibly even resulting in human extinction. Given the high stakes, it is important to analyze ASI risk and factor the risk into decisions related to ASI research and development. This paper presents a graphical model of major pathways to ASI catastrophe, focusing on ASI created via recursive self-improvement. The model uses the established risk and decision analysis modelling paradigms of fault trees and influence diagrams in order to depict combinations of events and conditions that could lead to AI catastrophe, as well as intervention options that could decrease risks. The events and conditions include select aspects of the ASI itself as well as the human process of ASI research, development and management. Model structure is derived from published literature on ASI risk. The model offers a foundation for rigorous quantitative evaluation and decision-making on the long-term risk of ASI catastrophe.

Keywords:

• Artificial superintelligence (ASI)
• catastrophe scenario pathway
• fault tree
• influence diagram

Acknowledgments

Thanks to Daniel Dewey, Nate Soares, Luke Muehlhauser, Miles Brundage, Kaj Sotala, Roman Yampolskiy, Eliezer Yudkowsky, Carl Shulman, Jeff Alstott, Steve Omohundro, Mark Waser, and two anonymous reviewers for comments on an earlier version of this paper, and to Stuart Armstrong and Anders Sandberg for helpful background discussion. Any remaining errors are the responsibility of the authors. Work on this paper was supported in part by a grant from the Future of Life Institute Fund. Any opinions, findings or recommendations in this document are those of the authors and do not necessarily reflect views of the Global Catastrophic Risk Institute, the Future of Life Institute, nor of others.

Funding

This work was supported by the Future of Life Institute Fund [2015-143911].

Notes

1. Eden, Moor, Soraker, and Steinhart (2012) present a range of perspectives on this controversy and related issues.

2. Seed AI and recursive self-improvement are often used specifically to improvements in software (e.g. Bostrom, 2014), but we use the term more generally in deference to the possibility of improvements in hardware (e.g. Sotala & Yampolskiy, 2015).

3. Throughout the paper, we will for the sake of brevity often use ‘catastrophe’ as shorthand for ‘major global catastrophe’.

4. On the moral importance of global catastrophes, see for example Matheny (2007), Bostrom (2013), Beckstead (2013).

5. Confinement and AI enforcement are types of containment, with containment being any measure to restrict the AI’s ability to affect the rest of the world such that the AI does not gain decisive strategic advantage. Other types of containment are measures taken by the rest of society and not built into the AI itself. We do not model these other types of containment as risk reduction interventions.

6. Goertzel and Pitt (2012) argue that, under some circumstances, connecting the AI to the world could increase safety. Even in such circumstances, keeping the AI disconnected would still qualify as confinement: confinement does not necessarily increase safety.

7. The use of white text on black background is not a standard influence diagram visual convention. We use it to further distinguish decision nodes from random-variable nodes in our figures.

8. The model can be adapted for analyzing safe ASI by switching the layer 2 right node to ‘ASI actions are safe’; much of the lower layer modelling (discussed below) will be similar.

9. The term AGI can refer to more than just seed AI. Indeed, an ASI built from a seed AGI would likely also be an AGI, as the AI is unlikely to lose general intelligence as it undergoes recursive self-improvement.

10. One might argue that a ‘hardware quantity’ soft takeoff is unlikely on grounds that if an AI is able to access more hardware than its designers intended, it would likely be able to access much or all of the entire world’s hardware all at once, causing a hard takeoff. Whilst quantifying scenario probabilities is beyond the scope of this paper, such probabilities can be plugged directly into the model.

11. For discussion of this and related ideas, see for example Yampolskiy and Fox (2013), Omohundro (2012, 2014), Soares and Fallenstein (2014), and Russell, Dewey, and Tegmark (2015). Details of failure modes for these ASI goal safety measures could be modelled in extensions of Figure 3.

12. On the precautionary principle in the context of catastrophic risk, see for example Posner (2004); Sunstein (2009).

13. Goal safety does not require the seed AI or subsequent pre-superintelligence AIs to have safe goals – thus included here are ideas like ‘coherent extrapolated volition’ (Yudkowsky, 2004) in which AI acquires safe goals during takeoff.

14. See also Soares (2015) for a characterisation of openness to training: ‘corrigibility’, which an AI system would possess ‘if it cooperates with what its creators regard as a corrective intervention, despite default incentives for rational agents to resist attempts to shut them down or modify their preferences’.