“Putting the humanity into inhuman systems: How human factors and ergonomics can be used to manage the risks associated with artificial general intelligence”, Salmon et al 2020

2020-salmon.pdf: “Putting the humanity into inhuman systems: How human factors and ergonomics can be used to manage the risks associated with artificial general intelligence”⁠, Paul M. Salmon, Tony Carden, Peter A. Hancock (2020-12-28):

The next generation of artificial intelligence, known as artificial general intelligence (AGI) could either revolutionize or destroy humanity. As the discipline which focuses on enhancing human health and wellbeing, human factors and ergonomics (HFE) has a crucial role to play in the conception, design, and operation of AGI systems. Despite this, there has been little examination as to how HFE can influence and direct this evolution. This study uses a hypothetical AGI system, Tegmark’s “Prometheus”, to frame the role of HFE in managing the risks associated with AGI. Fifteen categories of HFE method are identified and their potential role in AGI system design is considered. The findings suggest that all categories of HFE method can contribute to AGI system design; however, areas where certain methods require extension are identified. It is concluded that HFE can and should contribute to AGI system design and immediate effort is required to facilitate this goal. In closing, we explicate some of the work required to embed HFE in wider multi-disciplinary efforts aiming to create safe and efficient AGI systems.

“Counterproductive Altruism: The Other Heavy Tail”, Kokotajlo & Oprea 2020

2020-kokotajlo.pdf: “Counterproductive Altruism: The Other Heavy Tail”⁠, Daniel Kokotajlo, Alexandra Oprea (2020-05-30; statistics /​ ​order; backlinks):

First, we argue that the appeal of effective altruism (henceforth, EA) depends substantially on a certain empirical premise we call the Heavy Tail Hypothesis (HTH), which characterizes the probability distribution of opportunities for doing good. Roughly, the HTH implies that the best causes, interventions, or charities produce orders of magnitude greater good than the average ones, constituting a substantial portion of the total amount of good caused by altruistic interventions. Next, we canvass arguments EAs have given for the existence of a positive (or “right”) heavy tail and argue that they can also apply in support of a negative (or “left”) heavy tail where counterproductive interventions do orders of magnitude more harm than ineffective or moderately harmful ones. Incorporating the other heavy tail of the distribution has important implications for the core activities of EA: effectiveness research, cause prioritization, and the assessment of altruistic interventions. It also informs the debate surrounding the institutional critique of EA.

“Multiverse-wide Cooperation via Correlated Decision Making”, Oesterheld 2018

“Multiverse-wide Cooperation via Correlated Decision Making”⁠, Caspar Oesterheld (2018-01-29; backlinks):

Some decision theorists argue that when playing a prisoner’s dilemma-type game against a sufficiently similar opponent, we should cooperate to make it more likely that our opponent also cooperates. This idea, which Hofstadter calls superrationality, has strong implications when combined with the insight from modern physics that we probably live in a large universe or multiverse of some sort. If we care about what happens in civilizations located elsewhere in the multiverse, we can superrationally cooperate with some of their inhabitants. That is, if we take their values into account, this makes it more likely that they do the same for us. In this paper, I attempt to assess the practical implications of this idea. I argue that to reap the full gains from trade, everyone should maximize the same impartially weighted sum of the utility functions of all collaborators. I also argue that we can obtain at least weak evidence about the content of these utility functions. In practice, the application of superrationality implies that we should promote causal cooperation, moral pluralism, moral reflection, and ensure that our descendants, who will be smarter and thus better at finding out how to benefit other superrationalists in the universe, engage in superrational cooperation.

“There’s No Fire Alarm for Artificial General Intelligence”, Yudkowsky 2017

“There’s No Fire Alarm for Artificial General Intelligence”⁠, Eliezer Yudkowsky (2017-10-13; ai /​ ​scaling; backlinks):

[Meditation on the problem of coordinating reaction to x-risks, and AI risks in particular. To quote Norbert Wiener:

Again and again I have heard the statement that learning machines cannot subject us to any new dangers, because we can turn them off when we feel like it. But can we? To turn a machine off effectively, we must be in possession of information as to whether the danger point has come. The mere fact that we have made the machine does not guarantee we shall have the proper information to do this.

A fire alarm, even if it is not 100% accurate, coordinates human reactions: it becomes permissible to leave the room and investigate, take precautions, and for everyone to evacuate the building. This is because we all agree that fires usually come with smoke and smoke can be objectively detected. But what is the fire alarm for AI? “AI is whatever we can’t do yet”, and whenever AI accomplishes a new feat, people will simply move the goalposts and say that that task turned out to be unexpectedly easy to solve. There is no agreement on what “imminent AGI” looks like. You can ask AI researchers, “how would the world look different if we were in fact heading towards AGI in the near future, the next decade or three?” and they are unable to answer. They do not know what is or is not a ringing alarm bell, the point at which everyone should start taking the prospect very seriously. It was not chess, it was not ImageNet classification, it was not Go…

AI so far resembles other technologies like airplanes or nuclear bombs where just years before, the physicists who would invent it, eminent physicists, and physicists in general, were highly uncertain or skeptical or outright convinced of their impossibility. This was because progress in nuclear physics looked much the same regardless of whether nuclear bombs were possible and impossible. There was large ineradicable uncertainty, which appears to have neutered any serious effort to prepare. And yet, these matters ought to be dealt with in advance. Things like nuclear bombs or AI should not just arrive with no one having done anything to prepare. Or consider pandemics. Those who tried to warn the world about coronavirus will find this essay eerily apt.]

Okay, let’s be blunt here. I don’t think most of the discourse about AGI being far away (or that it’s near) is being generated by models of future progress in machine learning. I don’t think we’re looking at wrong models; I think we’re looking at no models.

I was once at a conference…I got up in Q&A and said, “Okay, you’ve all told us that progress won’t be all that fast. But let’s be more concrete and specific. I’d like to know what’s the least impressive accomplishment that you are very confident cannot be done in the next two years.”

There was a silence.

Eventually, 2 people on the panel ventured replies, spoken in a rather more tentative tone than they’d been using to pronounce that AGI was decades out. They named “A robot puts away the dishes from a dishwasher without breaking them”, and Winograd schemas…A few months after that panel, there was unexpectedly a big breakthrough on Winograd schemas. The breakthrough didn’t crack 80%, so three cheers for wide credibility intervals with error margin, but I expect the predictor might be feeling slightly more nervous now with one year left to go…

But that’s not the point. The point is the silence that fell after my question, and that eventually I only got 2 replies, spoken in tentative tones. When I asked for concrete feats that were impossible in the next two years, I think that that’s when the luminaries on that panel switched to trying to build a mental model of future progress in machine learning, asking themselves what they could or couldn’t predict, what they knew or didn’t know. And to their credit, most of them did know their profession well enough to realize that forecasting future boundaries around a rapidly moving field is actually really hard, that nobody knows what will appear on arXiv next month, and that they needed to put wide credibility intervals with very generous upper bounds on how much progress might take place 24 months’ worth of arXiv papers later. (Also, Demis Hassabis was present, so they all knew that if they named something insufficiently impossible, Demis would have DeepMind go and do it.)

…When I observe that there’s no fire alarm for AGI, I’m not saying that there’s no possible equivalent of smoke appearing from under a door. What I’m saying rather is that the smoke under the door is always going to be arguable; it is not going to be a clear and undeniable and absolute sign of fire; and so there is never going to be a fire alarm producing common knowledge that action is now due and socially acceptable…There is never going to be a time before the end when you can look around nervously, and see that it is now clearly common knowledge that you can talk about AGI being imminent, and take action and exit the building in an orderly fashion, without fear of looking stupid or frightened.

“Singularity and Inevitable Doom”, Prinz 2013

2013-prinz.pdf: “Singularity and Inevitable Doom”⁠, Jesse Prinz

“Intelligence Explosion: Evidence and Import”, Muehlhauser & Salamon 2012

2012-muehlhauser.pdf: “Intelligence Explosion: Evidence and Import”⁠, Luke Muehlhauser, Anna Salamon

“Implications of an Anthropic Model of Evolution for Emergence of Complex Life and Intelligence”, Watson 2008

2008-watson.pdf: “Implications of an Anthropic Model of Evolution for Emergence of Complex Life and Intelligence”⁠, Andrew J. Watson (2008-02-21):

Structurally complex life and intelligence evolved late on Earth; models for the evolution of global temperature suggest that, due to the increasing solar luminosity, the future life span of the (eukaryote) biosphere will be “only” about another billion years, a short time compared to the ~4 Ga since life began. A simple stochastic model (Carter 1983) suggests that this timing might be governed by the necessity to pass a small number, n, of very difficult evolutionary steps, with n < 10 and a best guess of n = 4, in order for intelligent observers like ourselves to evolve.

Here I extend the model analysis to derive probability distributions for each step. Past steps should tend to be evenly spaced through Earth’s history, and this is consistent with identification of the steps with some of the major transitions in the evolution of life on Earth. A complementary approach, identifying the critical steps with major reorganizations in Earth’s biogeochemical cycles, suggests that the Archean-Proterozoic and Proterozoic-Phanerozoic transitions might be identified with critical steps.

The success of the model lends support to a “Rare Earth” hypothesis (Rare Earth: Why Complex Life Is Uncommon in the Universe, Ward & Brownlee, 2000): structurally complex life is separated from prokaryotes by several very unlikely steps and, hence, will be much less common than prokaryotes. Intelligence is one further unlikely step, so it is much less common still.

“Must Early Life Be Easy? The Rhythm of Major Evolutionary Transitions”, Hanson 1998

1998-hanson.pdf: “Must Early Life Be Easy? The Rhythm of Major Evolutionary Transitions”⁠, Robin Hanson (1998-09-23):

If we are not to conclude that most planets like Earth have evolved life as intelligent as we are, we must presume Earth is not random. This selection effect, however, also implies that the origin of life need not be as easy as the early appearance of life on Earth suggests. If a series of major evolutionary transitions were required to produce intelligent life, selection implies that a subset of these were “critical steps”, with durations that are similarly distributed. The time remaining from now until simple life is no longer possible on Earth must also be similarly distributed. I show how these results provide timing tests to constrain models of critical evolutionary transitions.

“Implications of the Copernican principle for our future prospects”, III 1993

1993-gott.pdf: “Implications of the Copernican principle for our future prospects”⁠, J. Richard Gott III (1993-05-27):

Making only the assumption that you are a random intelligent observer, limits for the total longevity of our species of 0.2 million to 8 million years can be derived at the 95% confidence level. Further consideration indicates that we are unlikely to colonize the Galaxy, and that we are likely to have a higher population than the median for intelligent species.

“The Logic of Failure [and Discussion]”, Dörner et al 1990

1990-dorner.pdf: “The Logic of Failure [and Discussion]”⁠, D. Dörner, P. Nixon, S. D. Rosen (1990-04-12):

Unlike other living creatures, humans can adapt to uncertainty. They can form hypotheses about situations marked by uncertainty and can anticipate their actions by planning. They can expect the unexpected and take precautions against it. In numerous experiments, we have investigated the manner in which humans deal with these demands. In these experiments, we used computer simulated scenarios representing, for example, a small town, ecological or economic systems or political systems such as a Third World country. Within these computer-simulated scenarios, the subjects had to look for information, plan actions, form hypotheses, etc.

“Metamagical Themas: Sanity and Survival”, Hofstadter 1985

1985-hofstadter-sanityandsurvival.pdf: “Metamagical Themas: Sanity and Survival”⁠, Douglas Hofstadter (backlinks)

“Metamagical Themas: Sanity and Survival”, Hofstadter 1985

1985-hofstadter-sanityandsurvival.pdf: “Metamagical Themas: Sanity and Survival”⁠, Douglas Hofstadter (backlinks)

“The anthropic principle and its implications for biological evolution”, Carter 1983

1983-carter.pdf: “The anthropic principle and its implications for biological evolution”⁠, B. Carter (1983):

In the form in which it was originally expounded⁠, the anthropic principle was presented as a warning to astrophysical and cosmological theorists of the risk of error in the interpretation of astronomical and cosmological information unless due account is taken of the biological restraints under which the information was acquired. However, the converse message is also valid: biological theorists also run the risk of error in the interpretation of the evolutionary record unless they take due heed of the astrophysical restraints under which evolution took place. After an introductory discussion of the ordinary (‘weak’) anthropic principle and of its more contestable (‘strong’) analogue, a new application of the former to the problem of the evolution of terrestrial life is presented. It is shown that the evidence suggests that the evolutionary chain included at least one but probably not more than two links that were highly improbable (a priori) in the available time interval.

“Rejoinder to Gray and Wolfe”, Pascal 1980

1980-pascal.pdf: “Rejoinder to Gray and Wolfe”⁠, Louis Pascal (1980; psychology; backlinks):

This rejoinder to J. Patrick Gray’s and Linda Wolfe’s “The Loving Parent Meets the Selfish Gene” (Inquiry, this issue), which in turn was in response to the author’s “Human Tragedy and Natural Selection” (Inquiry, Vol. 21, No. 4), briefly addresses their major objections and suggests that in many instances they have misunderstood the point of that paper.

They argue that many of the traits referred to are more cultural than genetic. That this is not the central issue is made clearer by stressing certain aspects of the view underlying the original article, chiefly concerning the extent of human irrationality and insensitivity.

“The loving parent meets the selfish gene”, Gray & Wolfe 1980

1980-gray.pdf: “The loving parent meets the selfish gene”⁠, J. Patrick Gray, Linda Wolfe (1980; psychology; backlinks):

In a recent Inquiry article Louis Pascal argues that the problem of massive starvation in the modern world is the result of a genetically-based human propensity to produce as many offspring as possible, regardless of ecological conditions.

In this paper biological and anthropological objections to Pascal’s thesis are discussed as well as the conclusions he draws from it. It is suggested that natural selection has produced humans who are flexible in their reproductive behavior in order to cope with rapidly changing environments.

The implications of both arguments for the population movement and the attempt to eliminate starvation are discussed.

“Human tragedy and natural selection”, Pascal 1978

1978-pascal.pdf: “Human tragedy and natural selection”⁠, Louis Pascal (1978; psychology; backlinks):

It is argued that too logical a mind is not favored by natural selection; rather, it is biologically useful to be able to rationalize away certain unpleasant aspects of reality.

In most cases this irrationality has to do either with our reproductive ideas or with our ways of viewing the future. In both cases the implications with regard to our ability to solve the current population growth/​​​resource shrinkage crisis are decidedly negative. Looked at from a slightly different perspective, this same phenomenon can be viewed as a selection among ideas and beliefs. Sometimes this selection is quite intense.

Indeed, there are certain ideas which the human mind simply cannot logically deal with, since if they should be true, then it is obvious that evolution will direct all its resources toward producing minds which will believe them to be false. An example of such an idea [global human population control/​​​overpopulation crisis] is discussed.

“Randomness and Mathematical Proof”, Chatin 1975

1975-chaitin.pdf: “Randomness and Mathematical Proof”⁠, Gregory Chatin (backlinks)

“Large Number Coincidences and the Anthropic Principle in Cosmology”, Carter 1974

1974-carter.pdf: “Large Number Coincidences and the Anthropic Principle in Cosmology”⁠, Brandon Carter (1974):

[Discussion of what Carter names the “anthropic principle”: “what we can expect to observe must be restricted by the conditions necessary for our presence as observers. (Although our situation is not necessarily central, it is inevitably privileged to some extent.)”

Carter appeals to this to explain various “large number coincidences” in particle physics & cosmology: various relationships between stars and proton mass constants, the Hubble expansion rate & age of the universe, radiation pressure allowing solid matter, the value of the gravitational constant—which have been used to justify exotic theories of physics with varying constants etc—are in fact implied by our existence.

While that doesn’t necessarily ‘explain’ the relationships, this basic statistical/​​​philosophical requirement greatly undermines the appeal of such theories, particularly given the plausibility of various kinds of multiverse and ensemble theories. There may be no particular reason for any specific ‘large number coincidence’ other than (a) it was possible and (b) it is required for our existence so we could not observe otherwise.]

“All You Ever Wanted to Know about MIRV and ICBM Calculations but Were Not Cleared to Ask”, Davis & Schilling 1973

1973-davis.pdf: “All You Ever Wanted to Know about MIRV and ICBM Calculations but Were Not Cleared to Ask”⁠, Lynn Etheridge Davis, Warner R. Schilling

“Can We Survive Technology?”, Neumann 1955

1955-vonneumann-canwesurvivetechnology.pdf: “Can We Survive Technology?”⁠, John von Neumann

“Don't Worry—It Can't Happen”, Harrington 1940

1940-sciam-harrington-nuclearweapons-dontworryitcanthappen.pdf: “Don't Worry—It Can't Happen”⁠, Jean Harrington (1940-05-01; ai /​ ​scaling; backlinks):

…Early last summer, in the midst of all this research, a chilly sensation began tingling up and down the spines of the experimenters. These extra neutrons that were being erupted—could they not in turn become involuntary bullets, flying from one exploding uranium nucleus into the heart of another, causing another fission which would itself cause still others? Wasn’t there a dangerous possibility that the uranium would at last become explosive? That the samples being bombarded in the laboratories at Columbia University, for example, might blow up the whole of New York City? To make matters more ominous, news of fission research from Germany, plentiful in the early part of 1939, mysteriously and abruptly stopped for some months. Had government censorship been placed on what might be a secret of military importance? The press and populace, getting wind of these possibly lethal goings-on, raised a hue and cry. Nothing daunted, however, the physicists worked on to find out whether or not they would be blown up, and the rest of us along with them. Now, a year after the original discovery, word comes from Paris that we don’t have to worry.

…With typical French—and scientific—caution, they added that this was perhaps true only for the particular conditions of their own experiment, which was carried out on a large mass of uranium under water. But most scientists agreed that it was very likely true in general.

…Readers made insomnious by “newspaper talk” of terrific atomic war weapons held in reserve by dictators may now get sleep.