/docs/xrisks/ Directory Listing



  • 2020-salmon.pdf: ⁠, Paul M. Salmon, Tony Carden, Peter A. Hancock (2020-12-28):

    The next generation of artificial intelligence, known as artificial general intelligence (AGI) could either revolutionize or destroy humanity. As the discipline which focuses on enhancing human health and wellbeing, human factors and ergonomics (HFE) has a crucial role to play in the conception, design, and operation of AGI systems. Despite this, there has been little examination as to how HFE can influence and direct this evolution. This study uses a hypothetical AGI system, Tegmark’s “Prometheus,” to frame the role of HFE in managing the risks associated with AGI. Fifteen categories of HFE method are identified and their potential role in AGI system design is considered. The findings suggest that all categories of HFE method can contribute to AGI system design; however, areas where certain methods require extension are identified. It is concluded that HFE can and should contribute to AGI system design and immediate effort is required to facilitate this goal. In closing, we explicate some of the work required to embed HFE in wider multi-disciplinary efforts aiming to create safe and efficient AGI systems.

  • 2020-kokotajlo.pdf: ⁠, Daniel Kokotajlo, Alexandra Oprea (2020-05-30; backlinks):

    First, we argue that the appeal of effective altruism (henceforth, EA) depends substantially on a certain empirical premise we call the Heavy Tail Hypothesis (HTH), which characterizes the probability distribution of opportunities for doing good. Roughly, the HTH implies that the best causes, interventions, or charities produce orders of magnitude greater good than the average ones, constituting a substantial portion of the total amount of good caused by altruistic interventions. Next, we canvass arguments EAs have given for the existence of a positive (or “right”) heavy tail and argue that they can also apply in support of a negative (or “left”) heavy tail where counterproductive interventions do orders of magnitude more harm than ineffective or moderately harmful ones. Incorporating the other heavy tail of the distribution has important implications for the core activities of EA: effectiveness research, cause prioritization, and the assessment of altruistic interventions. It also informs the debate surrounding the institutional critique of EA.

  • 2016-lipsitch.pdf: “Underprotection of Unpredictable Statistical Lives Compared to Predictable Ones” (backlinks)

  • 2013-prinz.pdf: “Singularity and Inevitable Doom”⁠, Jesse Prinz

  • 2012-muehlhauser.pdf: “Intelligence Explosion: Evidence and Import”⁠, Luke Muehlhauser, Anna Salamon

  • 2008-watson.pdf: ⁠, Andrew J. Watson (2008-02-21; backlinks):

    Structurally complex life and intelligence evolved late on Earth; models for the evolution of global temperature suggest that, due to the increasing solar luminosity, the future life span of the (eukaryote) biosphere will be “only” about another billion years, a short time compared to the ~4 Ga since life began. A simple stochastic model () suggests that this timing might be governed by the necessity to pass a small number, n, of very difficult evolutionary steps, with n < 10 and a best guess of n = 4, in order for intelligent observers like ourselves to evolve.

    Here I extend the model analysis to derive probability distributions for each step. Past steps should tend to be evenly spaced through Earth’s history, and this is consistent with identification of the steps with some of the major transitions in the evolution of life on Earth. A complementary approach, identifying the critical steps with major reorganizations in Earth’s biogeochemical cycles, suggests that the Archean-Proterozoic and Proterozoic-Phanerozoic transitions might be identified with critical steps.

    The success of the model lends support to a “Rare Earth” hypothesis (Rare Earth: Why Complex Life Is Uncommon in the Universe, Ward and Brownlee, 2000): structurally complex life is separated from prokaryotes by several very unlikely steps and, hence, will be much less common than prokaryotes. Intelligence is one further unlikely step, so it is much less common still.

  • 1998-hanson.pdf: ⁠, Robin Hanson (1998-09-23; backlinks):

    If we are not to conclude that most planets like Earth have evolved life as intelligent as we are, we must presume Earth is not random. This selection effect, however, also implies that the origin of life need not be as easy as the early appearance of life on Earth suggests. If a series of major evolutionary transitions were required to produce intelligent life, selection implies that a subset of these were “critical steps”, with durations that are similarly distributed. The time remaining from now until simple life is no longer possible on Earth must also be similarly distributed. I show how these results provide timing tests to constrain models of critical evolutionary transitions.

  • 1993-gott.pdf: ⁠, J. Richard Gott III (1993-05-27):

    Making only the assumption that you are a random intelligent observer, limits for the total longevity of our species of 0.2 million to 8 million years can be derived at the 95% confidence level. Further consideration indicates that we are unlikely to colonize the Galaxy, and that we are likely to have a higher population than the median for intelligent species.

  • 1990-dorner.pdf: ⁠, D. Dörner, P. Nixon, S. D. Rosen (1990-04-12):

    Unlike other living creatures, humans can adapt to uncertainty. They can form hypotheses about situations marked by uncertainty and can anticipate their actions by planning. They can expect the unexpected and take precautions against it. In numerous experiments, we have investigated the manner in which humans deal with these demands. In these experiments, we used computer simulated scenarios representing, for example, a small town, ecological or economic systems or political systems such as a Third World country. Within these computer-simulated scenarios, the subjects had to look for information, plan actions, form hypotheses, etc.

  • 1985-hofstadter-sanityandsurvival.pdf: “Metamagical Themas: Sanity and Survival”⁠, Douglas Hofstadter (backlinks)

  • 1985-hofstadter-sanityandsurvival.pdf: “Metamagical Themas: Sanity and Survival”⁠, Douglas Hofstadter (backlinks)

  • 1983-carter.pdf: ⁠, B. Carter (1983; backlinks):

    In the form in ⁠, the anthropic principle was presented as a warning to astrophysical and cosmological theorists of the risk of error in the interpretation of astronomical and cosmological information unless due account is taken of the biological restraints under which the information was acquired. However, the converse message is also valid: biological theorists also run the risk of error in the interpretation of the evolutionary record unless they take due heed of the astrophysical restraints under which evolution took place. After an introductory discussion of the ordinary (‘weak’) anthropic principle and of its more contestable (‘strong’) analogue, a new application of the former to the problem of the evolution of terrestrial life is presented. It is shown that the evidence suggests that the evolutionary chain included at least one but probably not more than two links that were highly improbable (a priori) in the available time interval.

  • 1974-carter.pdf: ⁠, Brandon Carter (1974; backlinks):

    [Discussion of what Carter names the “anthropic principle”: “what we can expect to observe must be restricted by the conditions necessary for our presence as observers. (Although our situation is not necessarily central, it is inevitably privileged to some extent.)”

    Carter appeals to this to explain various “large number coincidences” in particle physics & cosmology: various relationships between stars and proton mass constants, the Hubble expansion rate & age of the universe, radiation pressure allowing solid matter, the value of the gravitational constant—which have been used to justify exotic theories of physics with varying constants etc—are in fact implied by our existence.

    While that doesn’t necessarily ‘explain’ the relationships, this basic statistical/philosophical requirement greatly undermines the appeal of such theories, particularly given the plausibility of various kinds of multiverse and ensemble theories. There may be no particular reason for any specific ‘large number coincidence’ other than (a) it was possible and (b) it is required for our existence so we could not observe otherwise.]

  • 1973-davis.pdf: “All You Ever Wanted to Know about MIRV and ICBM Calculations but Were Not Cleared to Ask”⁠, Lynn Etheridge Davis, Warner R. Schilling

  • 1955-vonneumann-canwesurvivetechnology.pdf: “Can We Survive Technology?”⁠, John von Neumann

  • 1940-sciam-harrington-nuclearweapons-dontworryitcanthappen.pdf: ⁠, Jean Harrington (Scientific American) (1940-05-01; backlinks):

    …Early last summer, in the midst of all this research, a chilly sensation began tingling up and down the spines of the experimenters. These extra neutrons that were being erupted—could they not in turn become involuntary bullets, flying from one exploding uranium nucleus into the heart of another, causing another fission which would itself cause still others? Wasn’t there a dangerous possibility that the uranium would at last become explosive? That the samples being bombarded in the laboratories at Columbia University, for example, might blow up the whole of New York City? To make matters more ominous, news of fission research from Germany, plentiful in the early part of 1939, mysteriously and abruptly stopped for some months. Had government censorship been placed on what might be a secret of military importance? The press and populace, getting wind of these possibly lethal goings-on, raised a hue and cry. Nothing daunted, however, the physicists worked on to find out whether or not they would be blown up, and the rest of us along with them. Now, a year after the original discovery, word comes from Paris that we don’t have to worry.

    …With typical French—and scientific—caution, they added that this was perhaps true only for the particular conditions of their own experiment, which was carried out on a large mass of uranium under water. But most scientists agreed that it was very likely true in general.

    …Readers made insomnious by “newspaper talk” of terrific atomic war weapons held in reserve by dictators may now get sleep.

  • 2017-barrett.pdf (backlinks)

  • 2013-evansarieyh-doingenough.html (backlinks)

  • 2012-schneier-liarsandoutliers-ch13.pdf (backlinks)

  • 2011-05-10-givewell-holdenkarnofskyjaantallinn.doc (backlinks)

  • 1998-hamming.pdf (backlinks)