“DeepMind: The Podcast—Excerpts on AGI”, Kiely 2022

“DeepMind: The Podcast—Excerpts on AGI”⁠, William Kiely (2022-04-07; AI scaling⁠, AI safety⁠, statistics / prediction; similar):

DeepMind: The Podcast—Season 2 was released over the last ~1–2 months. The 2 episodes most relevant to AGI are:

• The road to AGI—DeepMind: The Podcast (S2, Ep5) and
• The promise of AI with Demis Hassabis—DeepMind: The Podcast (S2, Ep9)

…The road to AGI (S2, Ep5)

(Published February 15, 2022)

Shane Legg’s AI Timeline; Shane Legg (4:03):

If you go back 10–12 years ago the whole notion of AGI was lunatic fringe. People [in the field] would literally just roll their eyes and just walk away. […] [I had that happen] multiple times. I have met quite a few of them since. There have even been cases where some of these people have applied for jobs at DeepMind years later. But yeah, it was a field where you know there were little bits of progress happening here and there, but powerful AGI and rapid progress seemed like it was very, very far away. […] Every year it [the number of people who roll their eyes at the notion of AGI] becomes less.

Hannah Fry (5:02):

For over 20 years, Shane has been quietly making predictions of when he expects to see AGI.

Shane Legg (5:09):

I always felt that somewhere around 2030-ish it was about a 50–50 chance. I still feel that seems reasonable. If you look at the amazing progress in the last 10 years and you imagine in the next 10 years we have something comparable, maybe there’s some chance that we will have an AGI in a decade. And if not in a decade, well I don’t know, say 3 decades or so.

…Demis Hassabis (7:11):

So I think that the progress so far has been pretty phenomenal. I think that it’s [AGI] coming relatively soon in the next you know—I wouldn’t be super surprised—in the next decade or 2.

---

[on convergent instrumental drives creating emergence]

Hannah Fry (21:59):

I put this question about the difficulty of designing an all-powerful reward to David Silver.

David Silver (22:05):

I actually think this is just slightly off the mark—this question—in the sense that maybe we can put almost any reward into the system and if the environment’s complex enough amazing things will happen just in maximizing that reward. Maybe we don’t have to solve this “What’s the right thing for intelligence to really emerge at the end of it?” kind of question and instead embrace the fact that there are many forms of intelligence, each of which is optimizing for its own target. And it’s okay if we have AIs in the future some of which are trying to control satellites and some of which are trying to sail boats and some of which are trying to win games of chess and they may all come up with their own abilities in order to allow that intelligence to achieve its end as effectively as possible.

[…] (26:14)

But of course this is a hypothesis. I cannot offer any guarantee that reinforcement learning algorithms do exist which are powerful enough to just get all the way there. And yet the fact that if we can do it, it would provide a path all the way to AGI should be enough for us to try really really hard.

See Also:

• “DeepMind and Google: the battle to control artificial intelligence. Demis Hassabis founded a company to build the world’s most powerful AI. Then Google bought him out. Hal Hodson asks who is in charge”

• “The messy, secretive reality behind OpenAI’s bid to save the world: The AI moonshot was founded in the spirit of transparency. This is the inside story of how competitive pressure eroded that idealism”

• “Tick, tock, tick, tock… BING”

“Joan Rohlfing on How to Avoid Catastrophic Nuclear Blunders: The Interaction between Nuclear Weapons and Cybersecurity”, Wiblin & Rohlfing 2022

“Joan Rohlfing on how to avoid catastrophic nuclear blunders: The interaction between nuclear weapons and cybersecurity”⁠, Rob Wiblin, Joan Rohlfing (2022-03-29; tech; backlinks; similar):

Rob Wiblin: Yeah. I’m interested to talk for a second about the interaction between nuclear weapons and cybersecurity. I just started reading this book called Hacking the Bomb—which from the title sounds a little bit sensationalist, but actually I can recommend it, because it’s quite a serious and sober, more academic look at the issues of play here.

Joan Rohlfing: Is this Andrew Futter’s book?

RW: I think that’s right, yeah. What do you think people who are somewhat informed about this area don’t know or don’t recognize about the interaction between nuclear weapons and cybersecurity?

JR: Thanks for raising that. Andrew Futter is doing some really important work in this space. For me, a cyber hack of our nuclear weapons is one of the most likely pathways to nuclear use. I worry a lot about it. NTI has done some [2018] work on this space. In 2018, we convened a study group with senior former military officials, civilian government officials, and experts in the field to look at this issue of implications of cyber vulnerabilities of nuclear weapons.

In part, we were motivated to do that because we had been watching this space closely, and were very concerned about a report that was published by the Defense Department itself. The Defense Department has an advisory board called the Defense Science Board that undertook a study in the 2013 timeframe, and published a report that basically says—and I’m paraphrasing the top-level recommendation—that all of our military operational systems are vulnerable to cyber attacks.

If you think about it, this is logical. All of our military forces have thousands of digital components—not all of those digital components come out of secure foundries, so we may be baking into our military systems faulty, compromised components. Then we also know that even if you have systems that aren’t directly connected to the internet air gap⁠, there are a lot of ways they can be compromised by an adversary. We saw how that might work with the cyberattack on Iranian centrifuges—which were not connected to the internet and nevertheless had a massive failure because of the introduction of a cyberattack.

The upshot of the report is that we have to assume that our nuclear forces may already be compromised, and that there is no technical solution. That’s the other chilling part of that story: this is not something you can just patch and be done, and you’re fine. It forces us to rethink: if this is true and we can’t have confidence in the system that it’s going to work as designed—that it’s not compromised—then what kind of policy changes do we need to be thinking about to counter what we can’t manage and that there’s no technical fix to? So this is a very real problem.

By the way, Andrew Futter, whose book you just referred to, was one of the participants in the 2018 NTI study on cyber-vulnerabilities of nuclear systems, which also included a former Head of US Strategic Command⁠, and former Vice Chairman of the Joint Chiefs of Staff⁠. There is consensus that this is a very substantial problem.

“Synthetic Fat from Petroleum As a Resilient Food for Global Catastrophes: Preliminary Techno-economic Assessment and Technology Roadmap”, Martínez et al 2022

2021-martinez.pdf: “Synthetic fat from petroleum as a resilient food for global catastrophes: Preliminary techno-economic assessment and technology roadmap”⁠, Juan B. García Martínez, Kyle A. Alvarado, David C. Denkenberger (2022-01; science⁠, tech; similar):

• Synthetic fat has substantial potential as resilient food for catastrophes.
• Fat requirements of the global population could be fulfilled in 1–2 years.
• 16–100% of the global fat requirements could be fulfilled by the end of the first year.
• The product would be affordable at an expected retail cost between US$3–$9/​kg.
• A roadmap for the development of synthetic fat as food for catastrophes is proposed.

Human civilization’s food production system is unprepared for global catastrophic risks (GCRs). Catastrophes capable of abruptly transforming global climate such as supervolcanic eruption⁠, asteroid/​comet impact or nuclear winter⁠, which could completely collapse the agricultural system. Responding by producing resilient foods requiring little to no sunlight is more cost effective than increasing food stockpiles, given the long duration of these scenarios (6–10 years).

This preliminary techno-economic assessment uncovers substantial potential for synthetic fat from petroleum as a resilient food source in the case of an abrupt sunlight reduction catastrophe, the most severe food shock scenario. To this end, the following are roughly quantified based on literature data: global production potential, capital and operating expenditures, material and energy requirements, ramp-up rates and retail prices. Potential resource bottlenecks are reviewed.

Synthetic fat production capacity would be slower to ramp up compared to low-tech food production alternatives, but provides the fat macronutrient, largely absent from these. Using 24/​7 construction of facilities, 16–100% of global fat requirements could be fulfilled at the end of the first year, potentially taking up to 2 years to fully meet the requirements. Substantial uncertainty remains on several topics including production potential, capital expenditure, food safety, transferability of labor and equipment construction. A technology roadmap is proposed to address these concerns and develop the potential of synthetic fat as a catastrophe-resilient food.

[Keywords: global catastrophic risk, existential-risk⁠, resilient food, synthetic fat, food security, nuclear winter]

…Historical context: This work is inspired by the historic precedent found in World War 2 Germany. In 1939, during a fat shortage, a non-biological process was developed to convert byproducts of the conversion from coal to liquid fuels into edible fat for human consumption. This byproduct, known as “gatsch” or paraffin wax, is a waxy fraction of the Fischer-Tropsch liquid product containing mostly alkane compounds, also known as paraffinic hydrocarbons or paraffins. These can be subjected to a chemical reaction known as paraffin oxidation⁠, causing the rupture of the alkanes into “synthetic” fatty acids (SFAs). Most of these SFAs were dedicated to soap production, but a part was processed into human food. These were subjected to purification and esterification with glycerol to produce triglycerides that could be refined into a synthetic margarine-like product. This was known as butter aus kohle or “coal butter” (BIOS 1946; Asinger 1968; Frankenfeld 1968).

In normal conditions, fats of agrichemical origin are more economical to produce than SFAs due to the capital intensity of the latter. However, during the shortage conditions in WW2 Germany the “coal butter” was allegedly cheaper to produce than regular butter (Eagle Valley Enterprise 1946), presumably due to the very high price of butter in Germany at the time. Pricing data from 1939 indicate the large retailer price of margarine at 1.74 Reichsmark (RM) per kilogram (Statistichen Reichsamt 1939), which would be ~equivalent to 13 USD in 2020.

Production of edible synthetic fat was discontinued completely during the 1950s, but SFA production continued to develop. In 1959 the USSR decided to replace 40% of the natural fatty acids used for soap production via SFA synthesis (Zilch 1968). In 1978, over 500,000 t of SFAs were obtained via continuous paraffin oxidation processes in Eastern Europe (Fineberg 1979). However, SFA production eventually fell out of use due to the lower cost and higher quality of agrichemical fatty acids, and has not been economical for several decades (Anneken et al 2006).

…Over a thousand animal trials were performed to prove the fat was neither toxic nor irritant and could be successfully digested. Later, experiments were performed on 6,000 human subjects over 3 years, which showed the product to be a satisfactory substitute for natural fat (BIOS 1946). There are reports of people allegedly consuming the synthetic fat in considerable amounts for over a year with no ill effects (Frankenfeld 1968). The synthetic fat was used by the German soldiers fighting in the African campaign and on submarines. It was also used in heavy labor rations, food for prisoners of war and concentration camp inmates and in canteen meals in hospitals (Reith & Pelzer-Reith 2002). The taste and calorific value were reportedly similar to those of butter (Der Spiegel 1947).

“Results of a 2020 Survey on Reporting Requirements and Practices for Biocontainment Laboratory Accidents”, Manheim 2021

“Results of a 2020 Survey on Reporting Requirements and Practices for Biocontainment Laboratory Accidents”⁠, David B. Manheim (2021-11-24; law⁠, sociology; similar):

Biosafety laboratory accidents are a normal part of laboratory science, but the frequency of such accidents is unclear due to current reporting standards and processes.

To better understand accident reporting, a survey was created, with input from ABSA International⁠, which included a series of questions about standards, requirements, and likely motivations for reporting or nonreporting. A total of 60 biosafety officers completed the survey. Respondents reported working with more than 5,000 people in laboratories, including more than 40 biosafety level 3 or animal biosafety level 3 laboratories, which work with higher-risk pathogens. Most of the respondents were located in the United States, Canada, or New Zealand, or did not identify their location.

Notable results included that 97% of surveyed biosafety officers oversee laboratories that require reporting exposure to at least some pathogens. However, 63% relayed that the reports are not usually sent outside of the institution where they occurred. A slight majority (55%) stated that paper reports were used, with the rest reporting they used a variety of computer systems. Even in laboratories that used paper-based reporting systems, 67% relayed that these reports were used alongside, or entered into, a digital system. While 82% of these biosafety officers agreed that workers understood the importance of reporting for their own safety, 82% also agreed that a variety of disincentives prevent laboratory workers from reporting incidents, including concerns about job loss and loss of funding.

[Keywords: laboratory safety⁠, laboratory acquired infections⁠, reporting, legal requirements, risk analysis]

“Late-time Small Body Disruptions for Planetary Defense”, King et al 2021

2021-king.pdf: “Late-time small body disruptions for planetary defense”⁠, Patrick K. King, Megan Bruck Syal, David S. P. Dearborn, Robert Managan, J. Michael Owen, Cody Raskin et al (2021-11-01; Radiance; similar):

• We present a fiducial simulation of the nuclear disruption of a small body.
• We simulate the orbits of the fragments for several plausible impact scenarios.
• We find that nuclear disruption is an effective strategy for late-time scenarios.
• Disruption can reduce the impacting mass to ~1% by one month before impact.

Diverting hazardous small bodies on impact trajectories with the Earth can in some circumstances be impossible without risking disrupting them. Disruption is a much more difficult planetary defense scenario to assess, being linked both to the response of the body to shock loading and the much more complicated gravitational dynamics of the fragments in the solar system relative to pure deflection scenarios.

In this work we present a new simulation suite built on n-body gravitational methods that solves fragment orbits in the full gravitational system without recourse to more approximate methods.

We assess the accuracy of our simulations and the simplifying assumptions we adopt to make the system tractable, and then discuss in more detail several specific, plausible planetary defense scenarios based on real close encounters. We find that disruption can be a very effective planetary defense strategy even for very late (sub-year) interventions, and should be considered an effective backup strategy should preferred methods, which require long warning times, fail.

[Keywords: planetary defense, catastrophic disruption, potentially hazardous asteroids (PHAs), celestial mechanics, n-body simulations]

“A Global Nucleic Acid Observatory for Biodefense and Planetary Health”, Consortium 2021

“A Global Nucleic Acid Observatory for Biodefense and Planetary Health”⁠, The Nucleic Acid Observatory Consortium (2021-08-05; genetics / sequencing; similar):

The spread of pandemic viruses and invasive species can be catastrophic for human societies and natural ecosystems. SARS-CoV-2 demonstrated that the speed of our response is critical, as each day of delay permitted exponential growth and dispersion of the virus. Here we propose a global Nucleic Acid Observatory (NAO) to monitor the relative frequency of everything biological through comprehensive metagenomic sequencing of waterways and wastewater. By searching for divergences from historical baseline frequencies at sites throughout the world, NAO could detect any virus or invasive organism undergoing exponential growth whose nucleic acids end up in the water, even those previously unknown to science. Continuously monitoring nucleic acid diversity would provide us with universal early warning, obviate subtle bioweapons, and generate a wealth of sequence data sufficient to transform ecology, microbiology, and conservation.

We call for the immediate construction of a global NAO to defend and illuminate planetary health.

“Objective Robustness in Deep Reinforcement Learning”, Koch et al 2021

“Objective Robustness in Deep Reinforcement Learning”⁠, Jack Koch, Lauro Langosco, Jacob Pfau, James Le, Lee Sharkey (2021-05-28; AI safety; backlinks; similar):

We study objective robustness failures, a type of out-of-distribution robustness failure in reinforcement learning (RL). Objective robustness failures occur when an RL agent retains its capabilities out-of-distribution yet pursues the wrong objective. This kind of failure presents different risks than the robustness problems usually considered in the literature, since it involves agents that leverage their capabilities to pursue the wrong objective rather than simply failing to do anything useful. We provide the first explicit empirical demonstrations of objective robustness failures and present a partial characterization of its causes.

“Putting the Humanity into Inhuman Systems: How Human Factors and Ergonomics Can Be Used to Manage the Risks Associated With Artificial General Intelligence”, Salmon et al 2020

2020-salmon.pdf: “Putting the humanity into inhuman systems: How human factors and ergonomics can be used to manage the risks associated with artificial general intelligence”⁠, Paul M. Salmon, Tony Carden, Peter A. Hancock (2020-12-28; similar):

The next generation of artificial intelligence, known as artificial general intelligence (AGI) could either revolutionize or destroy humanity. As the discipline which focuses on enhancing human health and wellbeing, human factors and ergonomics (HFE) has a crucial role to play in the conception, design, and operation of AGI systems. Despite this, there has been little examination as to how HFE can influence and direct this evolution. This study uses a hypothetical AGI system, Tegmark’s “Prometheus”, to frame the role of HFE in managing the risks associated with AGI. Fifteen categories of HFE method are identified and their potential role in AGI system design is considered. The findings suggest that all categories of HFE method can contribute to AGI system design; however, areas where certain methods require extension are identified. It is concluded that HFE can and should contribute to AGI system design and immediate effort is required to facilitate this goal. In closing, we explicate some of the work required to embed HFE in wider multi-disciplinary efforts aiming to create safe and efficient AGI systems.

“The Timing of Evolutionary Transitions Suggests Intelligent Life Is Rare”, Snyder-Beattie et al 2020

“The Timing of Evolutionary Transitions Suggests Intelligent Life Is Rare”⁠, Andrew E. Snyder-Beattie, Anders Sandberg⁠, K. Eric Drexler, Michael B. Bonsall (2020-11-19; similar):

It is unknown how abundant extraterrestrial life is, or whether such life might be complex or intelligent. On Earth, the emergence of complex intelligent life required a preceding series of evolutionary transitions such as abiogenesis, eukaryogenesis, and the evolution of sexual reproduction, multicellularity, and intelligence itself. Some of these transitions could have been extraordinarily improbable, even in conducive environments. The emergence of intelligent life late in Earth’s lifetime is thought to be evidence for a handful of rare evolutionary transitions, but the timing of other evolutionary transitions in the fossil record is yet to be analyzed in a similar framework.

Using a simplified Bayesian model that combines uninformative priors and the timing of evolutionary transitions, we demonstrate that expected evolutionary transition times likely exceed the lifetime of Earth, perhaps by many orders of magnitude. Our results corroborate the original argument suggested by Brandon Carter that intelligent life in the Universe is exceptionally rare, assuming that intelligent life elsewhere requires analogous evolutionary transitions. Arriving at the opposite conclusion would require exceptionally conservative priors, evidence for much earlier transitions, multiple instances of transitions, or an alternative model that can explain why evolutionary transitions took hundreds of millions of years without appealing to rare chance events.

Although the model is simple, it provides an initial basis for evaluating how varying biological assumptions and fossil record data impact the probability of evolving intelligent life, and also provides a number of testable predictions, such as that some biological paradoxes will remain unresolved and that planets orbiting M dwarf stars are uninhabitable.

[Previous work:

• Carter 1983, “The anthropic principle and its implications for biological evolution”
• Hanson 1998, “Must Early Life Be Easy? The Rhythm of Major Evolutionary Transitions”
• Carter 2008, “Five-step or six-step scenario for evolution?”
• Watson 2008, “Implications of an anthropic model of evolution for emergence of complex life and intelligence”
• McCabe & Lucas 2010, “On the origin and evolution of life in the Galaxy”
• Spiegel & Turner 2012, “Bayesian analysis of the astrobiological implications of life’s early emergence on Earth”]

“Possible Takeaways from the Coronavirus Pandemic for Slow AI Takeoff”, Vika 2020

“Possible takeaways from the coronavirus pandemic for slow AI takeoff”⁠, Vika (2020-05-31; sociology; backlinks; similar):

As the COVID-19 pandemic unfolds, we can draw lessons from it for managing future global risks, such as other pandemics, climate change, and risks from advanced AI…A key element in AI risk scenarios is the speed of takeoff—whether advanced AI is developed gradually or suddenly…slow AI takeoff is more likely than fast takeoff, but is not necessarily easier to manage, since it poses different challenges, such as large-scale coordination. This post expands on this point by examining some parallels between the coronavirus pandemic and a slow takeoff scenario. The upsides of slow takeoff include the ability to learn from experience, act on warning signs, and reach a timely consensus that there is a serious problem. I would argue that the COVID-19 pandemic had these properties, but most of the world’s institutions did not take advantage of them. This suggests that, unless our institutions improve, we should not expect the slow AI takeoff scenario to have a good default outcome.

1. Learning from experience… [none]
2. Warnings signs… [many]
3. Consensus on the problem… [none]

We can hope that the transformative technological change involved in the slow takeoff scenario will also help create more competent institutions without these weaknesses. We might expect that institutions unable to adapt to the fast pace of change will be replaced by more competent ones. However, we could also see an increasingly chaotic world where institutions fail to adapt without better institutions being formed quickly enough to replace them. Success in the slow takeoff scenario depends on institutional competence and large-scale coordination. Unless more competent institutions are in place by the time general AI arrives, it is not clear to me that slow takeoff would be much safer than fast takeoff.

“Counterproductive Altruism: The Other Heavy Tail”, Kokotajlo & Oprea 2020

2020-kokotajlo.pdf: “Counterproductive Altruism: The Other Heavy Tail”⁠, Daniel Kokotajlo, Alexandra Oprea (2020-05-30; order statistics; backlinks; similar):

First, we argue that the appeal of effective altruism (henceforth, EA) depends substantially on a certain empirical premise we call the Heavy Tail Hypothesis (HTH), which characterizes the probability distribution of opportunities for doing good. Roughly, the HTH implies that the best causes, interventions, or charities produce orders of magnitude greater good than the average ones, constituting a substantial portion of the total amount of good caused by altruistic interventions. Next, we canvass arguments EAs have given for the existence of a positive (or “right”) heavy tail and argue that they can also apply in support of a negative (or “left”) heavy tail where counterproductive interventions do orders of magnitude more harm than ineffective or moderately harmful ones. Incorporating the other heavy tail of the distribution has important implications for the core activities of EA: effectiveness research, cause prioritization, and the assessment of altruistic interventions. It also informs the debate surrounding the institutional critique of EA.

“2019 AI Alignment Literature Review and Charity Comparison”, Larks 2019

“2019 AI Alignment Literature Review and Charity Comparison”⁠, Larks (2019-12-18; neural net⁠, AI safety; similar):

As in 2016⁠, 2017 and 2018⁠, I have attempted to review the research that has been produced by various organisations working on AI safety, to help potential donors gain a better understanding of the landscape. This is a similar role to that which GiveWell performs for global health charities, and somewhat similar to a securities analyst with regards to possible investments. My aim is basically to judge the output of each organisation in 2019 and compare it to their budget. This should give a sense of the organisations’ average cost-effectiveness. We can also compare their financial reserves to their 2019 budgets to get a sense of urgency.

…Here are the un-scientifically-chosen hashtags: Agent Foundations · AI Theory · Amplification · Careers · CIRL · Decision Theory · Ethical Theory · Forecasting · Introduction · Misc · ML safety · Other Xrisk · Overview · Philosophy · Politics · RL · Security · Short-term · Strategy.

• Research organisations reviewed: FHI (The Future of Humanity Institute) · CHAI (The Center for Human-Aligned AI) · MIRI (The Machine Intelligence Research Institute) · GCRI (The Global Catastrophic Risks Institute) · CSER (The Center for the Study of Existential Risk) · Ought · OpenAI · Google DeepMind · AI Safety camp · FLI (The Future of Life Institute) · AI Impacts · GPI (The Global Priorities Institute) · FRI (The Foundational Research Institute) · Median Group · CSET (The Center for Security and Emerging Technology) · Leverhulme Center for the Future of Intelligence · BERI (The Berkeley Existential Risk Initiative) · AI Pulse
• Capital Allocators reviewed: LTFF (Long-term future fund) · OpenPhil (The Open Philanthropy Project)

…The size of the field continues to grow, both in terms of funding and researchers. Both make it increasingly hard for individual donors. I’ve attempted to subjectively weigh the productivity of the different organisations against the resources they used to generate that output, and donate accordingly.

“DeepMind and Google: the Battle to Control Artificial Intelligence. Demis Hassabis Founded a Company to Build the World’s Most Powerful AI. Then Google Bought Him Out. Hal Hodson Asks Who Is in Charge”, Hodson 2019

“DeepMind and Google: the battle to control artificial intelligence. Demis Hassabis founded a company to build the world’s most powerful AI. Then Google bought him out. Hal Hodson asks who is in charge”⁠, Hal Hodson (2019-03-01; law⁠, DM⁠, AI safety; backlinks; similar):

For many founders, this would be a happy ending. They could slow down, take a step back and spend more time with their money. For Hassabis⁠, the acquisition by Google was just another step in his pursuit of AGI. He had spent much of 2013 negotiating the terms of the deal. DeepMind would operate as a separate entity from its new parent. It would gain the benefits of being owned by Google, such as access to cash flow and computing power, without losing control.

…From the start, Hassabis has tried to protect DeepMind’s independence. He has always insisted that DeepMind remain in London. When Google bought the company in 2014, the question of control became more pressing. Hassabis didn’t need to sell DeepMind to Google. There was plenty of cash on hand and he had sketched out a business model in which the company would design games to fund research. Google’s financial heft was attractive, yet, like many founders, Hassabis was reluctant to hand over the company he had nurtured. As part of the deal, DeepMind created an arrangement that would prevent Google from unilaterally taking control of the company’s intellectual property. In the year leading up to acquisition, according to a person familiar with the transaction, both parties signed a contract called the Ethics and Safety Review Agreement. The agreement, previously unreported, was drawn up by senior barristers in London.

The Review Agreement puts control of DeepMind’s core AGI technology, whenever it may be created, in the hands of a governing panel known as the Ethics Board. Far from being a cosmetic concession from Google, the Ethics Board gives DeepMind solid legal backing to keep control of its most valuable and potentially most dangerous technology, according to the same source. The names of the panel members haven’t been made public, but another source close to both DeepMind and Google says that all 3 of DeepMind’s founders sit on the board. (DeepMind refused to answer a detailed set of questions about the Review Agreement but said that “ethics oversight and governance has been a priority for us from the earliest days.”)

See Also:

• “Organizational Update from OpenAI”

• “The messy, secretive reality behind OpenAI’s bid to save the world: The AI moonshot was founded in the spirit of transparency. This is the inside story of how competitive pressure eroded that idealism”

• “Anthropic raises $124 million to build more reliable, general AI systems”

• “Cooperative AI Foundation (CAIF)”

• “Goodbye 2010”

• “When will computer hardware match the human brain?”

“Forecasting Transformative AI: An Expert Survey”, Gruetzemacher et al 2019

“Forecasting Transformative AI: An Expert Survey”⁠, Ross Gruetzemacher, David Paradice, Kang Bok Lee (2019-01-24; neural net⁠, AI safety⁠, statistics / prediction⁠, survey; backlinks; similar):

Transformative AI technologies have the potential to reshape critical aspects of society in the near future. However, in order to properly prepare policy initiatives for the arrival of such technologies accurate forecasts and timelines are necessary. A survey was administered to attendees of three AI conferences during the summer of 2018 (ICML, IJCAI and the HLAI conference). The survey included questions for estimating AI capabilities over the next decade, questions for forecasting five scenarios of transformative AI and questions concerning the impact of computational resources in AI research. Respondents indicated a median of 21.5% of human tasks (ie. all tasks that humans are currently paid to do) can be feasibly automated now, and that this figure would rise to 40% in 5 years and 60% in 10 years. Median forecasts indicated a 50% probability of AI systems being capable of automating 90% of current human tasks in 25 years and 99% of current human tasks in 50 years. The conference of attendance was found to have a statistically-significant impact on all forecasts, with attendees of HLAI providing more optimistic timelines with less uncertainty. These findings suggest that AI experts expect major advances in AI technology to continue over the next decade to a degree that will likely have profound transformative impacts on society.

“Dissolving the Fermi Paradox”, Sandberg et al 2018

“Dissolving the Fermi Paradox”⁠, Anders Sandberg, Eric Drexler, Toby Ord (2018-06-06; philosophy; backlinks; similar):

The Fermi paradox is the conflict between an expectation of a high ex ante probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe. The expectation that the universe should be teeming with intelligent life is linked to models like the Drake equation, which suggest that even if the probability of intelligent life developing at a given site is small, the sheer multitude of possible sites should nonetheless yield a large number of potentially observable civilizations.

We show that this conflict arises from the use of Drake-like equations, which implicitly assume certainty regarding highly uncertain parameters. We examine these parameters, incorporating models of chemical and genetic transitions on paths to the origin of life, and show that extant scientific knowledge corresponds to uncertainties that span multiple orders of magnitude. This makes a stark difference. When the model is recast to represent realistic distributions of uncertainty, we find a substantial ex ante probability of there being no other intelligent life in our observable universe, and thus that there should be little surprise when we fail to detect any signs of it. This result dissolves the Fermi paradox, and in doing so removes any need to invoke speculative mechanisms by which civilizations would inevitably fail to have observable effects upon the universe.

“The Silurian Hypothesis: Would It Be Possible to Detect an Industrial Civilization in the Geological Record?”, Schmidt & Frank 2018

“The Silurian Hypothesis: Would it be possible to detect an industrial civilization in the geological record?”⁠, Gavin A. Schmidt, Adam Frank (2018-04-10; history⁠, tech; similar):

If an industrial civilization had existed on Earth many millions of years prior to our own era, what traces would it have left and would they be detectable today?

We summarize the likely geological fingerprint of the Anthropocene, and demonstrate that while clear, it will not differ greatly in many respects from other known events in the geological record.

We then propose tests that could plausibly distinguish an industrial cause from an otherwise naturally occurring climate event.

“The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities”, Lehman et al 2018

“The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities”⁠, Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J. Bentley et al (2018-03-09; AI⁠, RL / exploration⁠, AI safety; backlinks; similar):

Biological evolution provides a creative fount of complex and subtle adaptations, often surprising the scientists who discover them. However, because evolution is an algorithmic process that transcends the substrate in which it occurs, evolution’s creativity is not limited to nature. Indeed, many researchers in the field of digital evolution have observed their evolving algorithms and organisms subverting their intentions, exposing unrecognized bugs in their code, producing unexpected adaptations, or exhibiting outcomes uncannily convergent with ones in nature. Such stories routinely reveal creativity by evolution in these digital worlds, but they rarely fit into the standard scientific narrative. Instead they are often treated as mere obstacles to be overcome, rather than results that warrant study in their own right. The stories themselves are traded among researchers through oral tradition, but that mode of information transmission is inefficient and prone to error and outright loss. Moreover, the fact that these stories tend to be shared only among practitioners means that many natural scientists do not realize how interesting and lifelike digital organisms are and how natural their evolution can be. To our knowledge, no collection of such anecdotes has been published before. This paper is the crowd-sourced product of researchers in the fields of artificial life and evolutionary computation who have provided first-hand accounts of such cases. It thus serves as a written, fact-checked collection of scientifically important and even entertaining stories. In doing so we also present here substantial evidence that the existence and importance of evolutionary surprises extends beyond the natural world, and may indeed be an universal property of all complex evolving systems.

“Multiverse-wide Cooperation via Correlated Decision Making”, Oesterheld 2018

“Multiverse-wide Cooperation via Correlated Decision Making”⁠, Caspar Oesterheld (2018-01-29; backlinks; similar):

Some decision theorists argue that when playing a prisoner’s dilemma-type game against a sufficiently similar opponent, we should cooperate to make it more likely that our opponent also cooperates. This idea, which Hofstadter calls superrationality, has strong implications when combined with the insight from modern physics that we probably live in a large universe or multiverse of some sort. If we care about what happens in civilizations located elsewhere in the multiverse, we can superrationally cooperate with some of their inhabitants. That is, if we take their values into account, this makes it more likely that they do the same for us.

In this paper, I attempt to assess the practical implications of this idea. I argue that to reap the full gains from trade, everyone should maximize the same impartially weighted sum of the utility functions of all collaborators. I also argue that we can obtain at least weak evidence about the content of these utility functions. In practice, the application of superrationality implies that we should promote causal cooperation, moral pluralism, moral reflection, and ensure that our descendants, who will be smarter and thus better at finding out how to benefit other superrationalists in the universe, engage in superrational cooperation.

“There’s No Fire Alarm for Artificial General Intelligence”, Yudkowsky 2017

“There’s No Fire Alarm for Artificial General Intelligence”⁠, Eliezer Yudkowsky (2017-10-13; AI scaling⁠, AI safety; backlinks; similar):

[Meditation on the problem of coordinating reaction to x-risks, and AI risks in particular. To quote Norbert Wiener:

Again and again I have heard the statement that learning machines cannot subject us to any new dangers, because we can turn them off when we feel like it. But can we? To turn a machine off effectively, we must be in possession of information as to whether the danger point has come. The mere fact that we have made the machine does not guarantee we shall have the proper information to do this.

A fire alarm, even if it is not 100% accurate, coordinates human reactions: it becomes permissible to leave the room and investigate, take precautions, and for everyone to evacuate the building. This is because we all agree that fires usually come with smoke and smoke can be objectively detected. But what is the fire alarm for AI? “AI is whatever we can’t do yet”, and whenever AI accomplishes a new feat, people will simply move the goalposts and say that that task turned out to be unexpectedly easy to solve. There is no agreement on what “imminent AGI” looks like. You can ask AI researchers, “how would the world look different if we were in fact heading towards AGI in the near future, the next decade or three?” and they are unable to answer. They do not know what is or is not a ringing alarm bell, the point at which everyone should start taking the prospect very seriously. It was not chess, it was not ImageNet classification, it was not Go…

AI so far resembles other technologies like airplanes or nuclear bombs where just years before, the physicists who would invent it, eminent physicists, and physicists in general, were highly uncertain or skeptical or outright convinced of their impossibility. This was because progress in nuclear physics looked much the same regardless of whether nuclear bombs were possible and impossible. There was large ineradicable uncertainty, which appears to have neutered any serious effort to prepare. And yet, these matters ought to be dealt with in advance. Things like nuclear bombs or AI should not just arrive with no one having done anything to prepare. Or consider pandemics. Those who tried to warn the world about coronavirus will find this essay eerily apt.]

Okay, let’s be blunt here. I don’t think most of the discourse about AGI being far away (or that it’s near) is being generated by models of future progress in machine learning. I don’t think we’re looking at wrong models; I think we’re looking at no models.

I was once at a conference…I got up in Q&A and said, “Okay, you’ve all told us that progress won’t be all that fast. But let’s be more concrete and specific. I’d like to know what’s the least impressive accomplishment that you are very confident cannot be done in the next two years.”

There was a silence.

Eventually, 2 people on the panel ventured replies, spoken in a rather more tentative tone than they’d been using to pronounce that AGI was decades out. They named “A robot puts away the dishes from a dishwasher without breaking them”, and Winograd schemas…A few months after that panel, there was unexpectedly a big breakthrough on Winograd schemas. The breakthrough didn’t crack 80%, so three cheers for wide credibility intervals with error margin, but I expect the predictor might be feeling slightly more nervous now with one year left to go…

But that’s not the point. The point is the silence that fell after my question, and that eventually I only got 2 replies, spoken in tentative tones. When I asked for concrete feats that were impossible in the next two years, I think that that’s when the luminaries on that panel switched to trying to build a mental model of future progress in machine learning, asking themselves what they could or couldn’t predict, what they knew or didn’t know. And to their credit, most of them did know their profession well enough to realize that forecasting future boundaries around a rapidly moving field is actually really hard, that nobody knows what will appear on arXiv next month, and that they needed to put wide credibility intervals with very generous upper bounds on how much progress might take place 24 months’ worth of arXiv papers later. (Also, Demis Hassabis was present, so they all knew that if they named something insufficiently impossible, Demis would have DeepMind go and do it.)

…When I observe that there’s no fire alarm for AGI, I’m not saying that there’s no possible equivalent of smoke appearing from under a door. What I’m saying rather is that the smoke under the door is always going to be arguable; it is not going to be a clear and undeniable and absolute sign of fire; and so there is never going to be a fire alarm producing common knowledge that action is now due and socially acceptable…There is never going to be a time before the end when you can look around nervously, and see that it is now clearly common knowledge that you can talk about AGI being imminent, and take action and exit the building in an orderly fashion, without fear of looking stupid or frightened.

“Why Tool AIs Want to Be Agent AIs”, Branwen 2016

Tool-AI: “Why Tool AIs Want to Be Agent AIs”⁠, Gwern Branwen (2016-09-07; AI scaling / economics⁠, economics / automation⁠, insight porn⁠, AI safety⁠, transhumanism; backlinks; similar):

AIs limited to pure computation (Tool AIs) supporting humans, will be less intelligent, efficient, and economically valuable than more autonomous reinforcement-learning AIs (Agent AIs) who act on their own and meta-learn, because all problems are reinforcement-learning problems.

Autonomous AI systems (Agent AIs) trained using reinforcement learning can do harm when they take wrong actions, especially superintelligent Agent AIs. One solution would be to eliminate their agency by not giving AIs the ability to take actions, confining them to purely informational or inferential tasks such as classification or prediction (Tool AIs), and have all actions be approved & executed by humans, giving equivalently superintelligent results without the risk.

I argue that this is not an effective solution for two major reasons. First, because Agent AIs will by definition be better at actions than Tool AIs, giving an economic advantage. Secondly, because Agent AIs will be better at inference & learning than Tool AIs, and this is inherently due to their greater agency: the same algorithms which learn how to perform actions can be used to select important datapoints to learn inference over, how long to learn, how to more efficiently execute inference, how to design themselves, how to optimize hyperparameters, how to make use of external resources such as long-term memories or external software or large databases or the Internet, and how best to acquire new data.

All of these actions will result in Agent AIs more intelligent than Tool AIs, in addition to their greater economic competitiveness. Thus, Tool AIs will be inferior to Agent AIs in both actions and intelligence, implying use of Tool AIs is an even more highly unstable equilibrium than previously argued, as users of Agent AIs will be able to outcompete them on two dimensions (and not just one).

• Economic
• Intelligence
  • Actions for intelligence
    • Actions internal to a computation
    • Actions internal to training
    • Actions internal to data selection
    • Actions internal to NN design
    • Actions external to the agent
  • Overall
• Why You Shouldn’t Be A Tool
• See Also
• External Links

“Singularity and Inevitable Doom”, Prinz 2013

2013-prinz.pdf: “Singularity and Inevitable Doom”⁠, Jesse Prinz (2013-01-01)

“Metamagical Themas: Sanity and Survival”, Hofstadter 2012

1985-hofstadter: “Metamagical Themas: Sanity and Survival”⁠, Douglas Hofstadter (2012-04-16; ethics⁠, politics⁠, sociology; backlinks; similar):

3 essays by AI researcher Douglas Hofstadter exploring cooperation/​game theory/​‘superrationality’ in the context of the failure of political coordination to prevent global nuclear war

The following 3 essays were prepared from pages 737–780 of an ebook of Metamagical Themas: Questing for the Essence of Mind and Pattern (1985) by Douglas Hofstadter⁠, an anthology of articles & essays primarily published in Scientific American “between January 1981 and July 1983”. (I omit one entry in “Sanity and Survival”, the essay “The Tumult of Inner Voices, or, What is the Meaning of the Word ‘I’?”, which is unconnected to the other entries on cooperation/​decision theory⁠/​nuclear war.) All hyperlinks are my insertion.

They are interesting for introducing the idea of ‘superrationality’ in game theory⁠, an attempt to devise a decision theory/​algorithm for agents which can reach global utility maxima on problems like the prisoner’s dilemma even in the absence of coercion or communication which has partially inspired later decision theories like UDT or TDT⁠, linking decision theory to cooperation (eg. Oesterheld 2017) & existential risks (specifically, nuclear warfare), and one networking project⁠.

Note: to hide apparatus like the links, you can use reader-mode ().

• Sanity and Survival
  • Dilemmas for Superrational Thinkers, Leading Up to a Luring Lottery
    • Post Scriptum
  • Irrationality Is the Square Root of All Evil
    • Post Scriptum
  • The Tale of Happiton
    • Post Scriptum
    • Post Post Scriptum

“Bayesian Analysis of the Astrobiological Implications of Life’s Early Emergence on Earth”, Spiegel & Turner 2012

“Bayesian analysis of the astrobiological implications of life’s early emergence on Earth”⁠, David S. Spiegel, Edwin L. Turner (2012-01-10; biology; backlinks; similar):

Life arose on Earth sometime in the first few hundred million years after the young planet had cooled to the point that it could support water-based organisms on its surface. The early emergence of life on Earth has been taken as evidence that the probability of abiogenesis is high, if starting from young Earth-like conditions.

We revisit this argument quantitatively in a Bayesian statistical framework. By constructing a simple model of the probability of abiogenesis, we calculate a Bayesian estimate of its posterior probability, given the data that life emerged fairly early in Earth’s history and that, billions of years later, curious creatures noted this fact and considered its implications.

We find that, given only this very limited empirical information, the choice of Bayesian prior for the abiogenesis probability parameter has a dominant influence on the computed posterior probability. Although terrestrial life’s early emergence provides evidence that life might be abundant in the universe if early-Earth-like conditions are common, the evidence is inconclusive and indeed is consistent with an arbitrarily low intrinsic probability of abiogenesis for plausible uninformative priors⁠.

Finding a single case of life arising independently of our lineage (on Earth, elsewhere in the solar system, or on an extrasolar planet) would provide much stronger evidence that abiogenesis is not extremely rare in the universe.

“Intelligence Explosion: Evidence and Import”, Muehlhauser & Salamon 2012

2012-muehlhauser.pdf: “Intelligence Explosion: Evidence and Import”⁠, Luke Muehlhauser, Anna Salamon (2012-01-01)

“On the Origin and Evolution of Life in the Galaxy”, McCabe & Lucas 2011

“On the Origin and Evolution of Life in the Galaxy”⁠, Michael McCabe, Holly Lucas (2011-03-17; biology; backlinks; similar):

A simple stochastic model for evolution, based upon the need to pass a sequence of n critical steps (Carter 1983⁠, Watson 2008) is applied to both terrestrial and extraterrestrial origins of life.

In the former case, the time at which humans have emerged during the habitable period of the Earth suggests a value of n = 4. Progressively adding earlier evolutionary transitions (Maynard Smith and Szathmary, 1995) gives an optimum fit when n = 5, implying either that their initial transitions are not critical or that habitability began around 6 Ga ago.

The origin of life on Mars or elsewhere within the Solar System is excluded by the latter case and the simple anthropic argument is that extraterrestrial life is scarce in the Universe because it does not have time to evolve. Alternatively, the timescale can be extended if the migration of basic progenotic material to Earth is possible. If extra transitions are included in the model to allow for Earth migration, then the start of habitability needs to be even earlier than 6 Ga ago. Our present understanding of Galactic habitability and dynamics does not exclude this possibility.

We conclude that Galactic punctuated equilibrium (Cirkovic et al 2009), proposed as a way round the anthropic problem, is not the only way of making life more common in the Galaxy.

“Anthropic Shadow: Observation Selection Effects and Human Extinction Risks”, Ćirković et al 2010

“Anthropic Shadow: Observation Selection Effects and Human Extinction Risks”⁠, Milan M. Ćirković, Anders Sandberg, Nick Bostrom (2010-07-09; philosophy; similar):

We describe a substantial practical consequence of taking anthropic biases into account in deriving predictions for rare stochastic catastrophic events. The risks associated with catastrophes such as asteroidal/​cometary impacts, supervolcanic episodes, and explosions of supernovae/​gamma-ray bursts are based on their observed frequencies. As a result, the frequencies of catastrophes that destroy or are otherwise incompatible with the existence of observers are systematically underestimated. We describe the consequences of this anthropic bias for estimation of catastrophic risks, and suggest some directions for future work.

[Keywords: Anthropic principle⁠; astrobiology; existential risks; global catastrophes; impact hazard; natural hazards; risk management; selection effects; vacuum phase transition]

“Myopic Voters and Natural Disaster Policy”, Healy & Malhotra 2009

“Myopic Voters and Natural Disaster Policy”⁠, Andrew Healy, Neil Malhotra (2009-08-01; sociology; similar):

Do voters effectively hold elected officials accountable for policy decisions? Using data on natural disasters, government spending, and election returns, we show that voters reward the incumbent presidential party for delivering disaster relief spending, but not for investing in disaster preparedness spending. These inconsistencies distort the incentives of public officials, leading the government to underinvest in disaster preparedness, thereby causing substantial public welfare losses. We estimate that $1.4^$1.0₂₀₀₉ spent on preparedness is worth about $20.5^$15.0₂₀₀₉ in terms of the future damage it mitigates. By estimating both the determinants of policy decisions and the consequences of those policies, we provide more complete evidence about citizen competence and government accountability.

“The Physical Principles of Thermonuclear Explosives, Inertial Confinement Fusion, and the Quest for Fourth Generation Nuclear Weapons”, Gsponer & Hurni 2009

“The physical principles of thermonuclear explosives, inertial confinement fusion, and the quest for fourth generation nuclear weapons”⁠, Andre Gsponer, Jean-Pierre Hurni (2009-01-20; Radiance⁠, science; backlinks; similar):

This report is an assessment of the prospect of developing new (ie. fourth generation) nuclear weapons in the context of the Comprehensive Nuclear Test-Ban Treaty (CTBT) that was adopted by the UN General Assembly in 1996 and of the current moratorium on nuclear testing in effect in all nuclear-weapon States.

The conclusion stresses that considerable research is underway in all five nuclear-weapon States (as well as in several other major industrialized States such as Germany and Japan) on ICF and on many physical processes that provide the scientific basis necessary to develop fourth generation nuclear weapons. Substantial progress has been made in the past few years on all these processes, and the construction of large ICF microexplosion facilities in both nuclear-weapon and non-nuclear-weapon States is giving the arms race a fresh boost. The world runs the risk that certain countries will equip themselves directly with fourth generation nuclear weapons, bypassing the acquisition of previous generations of nuclear weapons.

“Probing the Improbable: Methodological Challenges for Risks With Low Probabilities and High Stakes”, Ord et al 2008

“Probing the Improbable: Methodological Challenges for Risks with Low Probabilities and High Stakes”⁠, Toby Ord, Rafaela Hillerbrand, Anders Sandberg (2008-10-30; math⁠, statistics / bias; backlinks; similar):

Some risks have extremely high stakes. For example, a worldwide pandemic or asteroid impact could potentially kill more than a billion people. Comfortingly, scientific calculations often put very low probabilities on the occurrence of such catastrophes. In this paper, we argue that there are important new methodological problems which arise when assessing global catastrophic risks and we focus on a problem regarding probability estimation.

When an expert provides a calculation of the probability of an outcome, they are really providing the probability of the outcome occurring, given that their argument is watertight. However, their argument may fail for a number of reasons such as a flaw in the underlying theory, a flaw in the modeling of the problem, or a mistake in the calculations. If the probability estimate given by an argument is dwarfed by the chance that the argument itself is flawed, then the estimate is suspect.

We develop this idea formally, explaining how it differs from the related distinctions of model and parameter uncertainty. Using the risk estimates from the Large Hadron Collider as a test case, we show how serious the problem can be when it comes to catastrophic risks and how best to address it.

“Implications of an Anthropic Model of Evolution for Emergence of Complex Life and Intelligence”, Watson 2008

2008-watson.pdf: “Implications of an Anthropic Model of Evolution for Emergence of Complex Life and Intelligence”⁠, Andrew J. Watson (2008-02-21; backlinks; similar):

Structurally complex life and intelligence evolved late on Earth; models for the evolution of global temperature suggest that, due to the increasing solar luminosity, the future life span of the (eukaryote) biosphere will be “only” about another billion years, a short time compared to the ~4 Ga since life began. A simple stochastic model (Carter 1983) suggests that this timing might be governed by the necessity to pass a small number, n, of very difficult evolutionary steps, with n < 10 and a best guess of n = 4, in order for intelligent observers like ourselves to evolve.

Here I extend the model analysis to derive probability distributions for each step. Past steps should tend to be evenly spaced through Earth’s history, and this is consistent with identification of the steps with some of the major transitions in the evolution of life on Earth. A complementary approach, identifying the critical steps with major reorganizations in Earth’s biogeochemical cycles, suggests that the Archean-Proterozoic and Proterozoic-Phanerozoic transitions might be identified with critical steps.

The success of the model lends support to a “Rare Earth” hypothesis (Rare Earth: Why Complex Life Is Uncommon in the Universe, Ward & Brownlee, 2000): structurally complex life is separated from prokaryotes by several very unlikely steps and, hence, will be much less common than prokaryotes. Intelligence is one further unlikely step, so it is much less common still.

“The Optimistic Thought Experiment”, Thiel 2008

“The Optimistic Thought Experiment”⁠, Peter Thiel (2008-01-28; economics; backlinks; similar):

[An interesting thought experiment to assess what must happen for an “optimistic” version of the future to unfold, and the possibility of an impending apocalypse and how that might lead to financial bubbles. The article is eye opening, depressing and fascinating. Peter argues that science in all of its form (nuclear weapons, biological catastrophes, etc) has vastly increased the probability of some form of apocalypse; betting on the apocalypse makes no sense so rational investors don’t do it; globalization is the anti-apocalypse bet; financial bubbles are bets on globalization; and the recent slate of financial bubbles, which he calls unprecedented in history, are related to the growing sense of impending doom.]

One would not have thought it possible for the internet bubble of the late 1990s, the greatest boom in the history of the world, to be replaced within 5 years by a real estate bubble of even greater magnitude and worse stupidity. Under more normal circumstances, one would not have thought that the same mistake could happen twice in the lifetimes of the people involved…

The most straightforward explanation begins with the view that all of these bubbles are not truly separate, but instead represent different facets of a single Great Boom of unprecedented size and duration. As with the earlier bubbles of the modern age, the Great Boom has been based on a similar story of globalization, told and retold in different ways—and so we have seen a rotating series of local booms and bubbles as investors price a globally unified world through the prism of different markets.

Nevertheless, this Great Boom is also very different from all previous bubbles. This time around, globalization either will succeed and humanity will achieve a degree of freedom and prosperity that can scarcely be imagined, or globalization will fail and capitalism or even humanity itself may come to an end. The real alternative to good globalization is world war. And because of the nature of today’s technology, such a war would be apocalyptic in the 21^st century. Because there is not much time left, the Great Boom, taken as a whole, either is not a bubble at all, or it is the final and greatest bubble in history…there is no good scenario for the world in which China fails.

…But because we do not know how our story of globalization will end, we do not yet know which it is. Let us return to our thought experiment. Let us assume that, in the event of successful globalization, a given business would be worth $100/​share, but that there is only an intermediate chance (say 1:10) of successful globalization. The other case is too terrible to consider. Theoretically, the share should be worth $10, but in every world where investors survive, it will be worth $100. Would it make sense to pay more than $10, and indeed any price up to $100? Whether in hope or desperation, the perceived lack of alternatives may push valuations to much greater extremes than in non-apocalyptic times.

“Five or Six Step Scenario for Evolution?”, Carter 2007

“Five or six step scenario for evolution?”⁠, Brandon Carter (2007-11-13; backlinks; similar):

The prediction that (due to the limited amount of hydrogen available as fuel in the Sun) the future duration of our favourable terrestrial environment will be short (compared with the present age of the Earth) has been interpreted as evidence for a hard step scenario. This means that some of the essential steps (such as the development of eukaryotes) in the evolution process leading to the ultimate emergence of intelligent life would have been hard, in the sense of being against the odds in the available time, so that they are unlikely to have been achieved in most of the earth-like planets that may one day be discovered in nearby extra-solar systems.

It was originally estimated that only one or two of the essential evolutionary steps had to have been hard in this sense, but it has become apparent that this figure may need upward revision, because recent studies of climatic instability suggest that the possible future duration of our biologically favourable environment may be shorter than had been supposed, only about 1 gigayear rather than 5.

On the basis of the statistical requirement of roughly equal spacing between hard steps, it is argued that the best fit with the fossil record is now obtainable by postulating the number of hard steps to be 5, if our evolution was exclusively terrestrial, or 6 if, as now seems very plausible, the first step occurred on Mars.

“Aberrant Innate Immune Response in Lethal Infection of Macaques With the 1918 Influenza Virus”, Kobasa et al 2007

“Aberrant innate immune response in lethal infection of macaques with the 1918 influenza virus”⁠, Darwyn Kobasa, Steven M. Jones, Kyoko Shinya, John C. Kash, John Copps, Hideki Ebihara, Yasuko Hatta et al (2007-01-18; similar):

The 1918 influenza pandemic was unusually severe, resulting in about 50 million deaths worldwide. The 1918 virus is also highly pathogenic in mice, and studies have identified a multigenic origin of this virulent phenotype in mice. However, these initial characterizations of the 1918 virus did not address the question of its pathogenic potential in primates.

Here we demonstrate that the 1918 virus caused a highly pathogenic respiratory infection in a cynomolgus macaque model that culminated in acute respiratory distress and a fatal outcome. Furthermore, infected animals mounted an immune response, characterized by dysregulation of the antiviral response, that was insufficient for protection, indicating that atypical host innate immune responses may contribute to lethality.

The ability of influenza viruses to modulate host immune responses, such as that demonstrated for the avian H5N1 influenza viruses⁠, may be a feature shared by the virulent influenza viruses.

“Must Early Life Be Easy? The Rhythm of Major Evolutionary Transitions”, Hanson 1998

1998-hanson.pdf: “Must Early Life Be Easy? The Rhythm of Major Evolutionary Transitions”⁠, Robin Hanson (1998-09-23; backlinks; similar):

If we are not to conclude that most planets like Earth have evolved life as intelligent as we are, we must presume Earth is not random. This selection effect, however, also implies that the origin of life need not be as easy as the early appearance of life on Earth suggests. If a series of major evolutionary transitions were required to produce intelligent life, selection implies that a subset of these were “critical steps”, with durations that are similarly distributed. The time remaining from now until simple life is no longer possible on Earth must also be similarly distributed. I show how these results provide timing tests to constrain models of critical evolutionary transitions.

“Mathematics on a Distant Planet”, Hamming 1998

1998-hamming.pdf: “Mathematics on a Distant Planet”⁠, Hamming (1998; math; backlinks)

“Implications of the Copernican Principle for Our Future Prospects”, III 1993

1993-gott.pdf: “Implications of the Copernican principle for our future prospects”⁠, J. Richard Gott III (1993-05-27; similar):

Making only the assumption that you are a random intelligent observer, limits for the total longevity of our species of 0.2 million to 8 million years can be derived at the 95% confidence level. Further consideration indicates that we are unlikely to colonize the Galaxy, and that we are likely to have a higher population than the median for intelligent species.

“The Logic of Failure [and Discussion]”, Dörner et al 1990

1990-dorner.pdf: “The Logic of Failure [and Discussion]”⁠, D. Dörner, P. Nixon, S. D. Rosen (1990-04-12; similar):

Unlike other living creatures, humans can adapt to uncertainty. They can form hypotheses about situations marked by uncertainty and can anticipate their actions by planning. They can expect the unexpected and take precautions against it. In numerous experiments, we have investigated the manner in which humans deal with these demands. In these experiments, we used computer simulated scenarios representing, for example, a small town, ecological or economic systems or political systems such as a Third World country. Within these computer-simulated scenarios, the subjects had to look for information, plan actions, form hypotheses, etc.

“Nuclear Fission: Reaction to the Discovery in 1939”, Badash et al 1986

1986-badash.pdf: “Nuclear Fission: Reaction to the Discovery in 1939”⁠, Lawrence Badash, Elizabeth Hodes, Adolph Tiddens (1986-06-01; similar):

A large body of literature exists on the scientific and political history of nuclear weapons. There is little, however, concerning the reaction in 1939 to news of the discovery of nuclear fission. This study is a detailed examination of worldwide views during the preceding 4 decades about “harnessing the energy of the atom”, a brief survey of the scientific accomplishments of 1939, a close look throughout that year at the thoughts, hopes, fears, and actions that fission inspired, primarily in America, and an analysis of why the discovery came as such a surprise, and why it generated relatively little moral or ethical introspection.

“Metamagical Themas: Sanity and Survival”, Hofstadter 1985

1985-hofstadter-sanityandsurvival.pdf: “Metamagical Themas: Sanity and Survival”⁠, Douglas Hofstadter (1985-01-01; backlinks)

“The Anthropic Principle and Its Implications for Biological Evolution”, Carter 1983

1983-carter.pdf: “The anthropic principle and its implications for biological evolution”⁠, B. Carter (1983; backlinks; similar):

In the form in which it was originally expounded⁠, the anthropic principle was presented as a warning to astrophysical and cosmological theorists of the risk of error in the interpretation of astronomical and cosmological information unless due account is taken of the biological restraints under which the information was acquired. However, the converse message is also valid: biological theorists also run the risk of error in the interpretation of the evolutionary record unless they take due heed of the astrophysical restraints under which evolution took place. After an introductory discussion of the ordinary (‘weak’) anthropic principle and of its more contestable (‘strong’) analogue, a new application of the former to the problem of the evolution of terrestrial life is presented. It is shown that the evidence suggests that the evolutionary chain included at least one but probably not more than two links that were highly improbable (a priori) in the available time interval.

“Rejoinder to Gray and Wolfe”, Pascal 1980

1980-pascal.pdf: “Rejoinder to Gray and Wolfe”⁠, Louis Pascal (1980; psychology; backlinks; similar):

This rejoinder to J. Patrick Gray’s and Linda Wolfe’s “The Loving Parent Meets the Selfish Gene” (Inquiry, this issue), which in turn was in response to the author’s “Human Tragedy and Natural Selection” (Inquiry, Vol. 21, No. 4), briefly addresses their major objections and suggests that in many instances they have misunderstood the point of that paper.

They argue that many of the traits referred to are more cultural than genetic. That this is not the central issue is made clearer by stressing certain aspects of the view underlying the original article, chiefly concerning the extent of human irrationality and insensitivity.

“The Loving Parent Meets the Selfish Gene”, Gray & Wolfe 1980

1980-gray.pdf: “The loving parent meets the selfish gene”⁠, J. Patrick Gray, Linda Wolfe (1980; psychology; backlinks; similar):

In a recent Inquiry article Louis Pascal argues that the problem of massive starvation in the modern world is the result of a genetically-based human propensity to produce as many offspring as possible, regardless of ecological conditions.

In this paper biological and anthropological objections to Pascal’s thesis are discussed as well as the conclusions he draws from it. It is suggested that natural selection has produced humans who are flexible in their reproductive behavior in order to cope with rapidly changing environments.

The implications of both arguments for the population movement and the attempt to eliminate starvation are discussed.

“Human Tragedy and Natural Selection”, Pascal 1978

1978-pascal.pdf: “Human tragedy and natural selection”⁠, Louis Pascal (1978; psychology; backlinks; similar):

It is argued that too logical a mind is not favored by natural selection; rather, it is biologically useful to be able to rationalize away certain unpleasant aspects of reality.

In most cases this irrationality has to do either with our reproductive ideas or with our ways of viewing the future. In both cases the implications with regard to our ability to solve the current population growth/​resource shrinkage crisis are decidedly negative. Looked at from a slightly different perspective, this same phenomenon can be viewed as a selection among ideas and beliefs. Sometimes this selection is quite intense.

Indeed, there are certain ideas which the human mind simply cannot logically deal with, since if they should be true, then it is obvious that evolution will direct all its resources toward producing minds which will believe them to be false. An example of such an idea [global human population control/​overpopulation crisis] is discussed.

“Randomness and Mathematical Proof”, Chatin 1975

1975-chaitin.pdf: “Randomness and Mathematical Proof”⁠, Gregory Chatin (1975-01-01; math; backlinks)

“Large Number Coincidences and the Anthropic Principle in Cosmology”, Carter 1974

1974-carter.pdf: “Large Number Coincidences and the Anthropic Principle in Cosmology”⁠, Brandon Carter (1974; backlinks; similar):

[Discussion of what Carter names the “anthropic principle”: “what we can expect to observe must be restricted by the conditions necessary for our presence as observers. (Although our situation is not necessarily central, it is inevitably privileged to some extent.)”

Carter appeals to this to explain various “large number coincidences” in particle physics & cosmology: various relationships between stars and proton mass constants, the Hubble expansion rate & age of the universe, radiation pressure allowing solid matter, the value of the gravitational constant—which have been used to justify exotic theories of physics with varying constants etc—are in fact implied by our existence.

While that doesn’t necessarily ‘explain’ the relationships, this basic statistical/​philosophical requirement greatly undermines the appeal of such theories, particularly given the plausibility of various kinds of multiverse and ensemble theories. There may be no particular reason for any specific ‘large number coincidence’ other than (a) it was possible and (b) it is required for our existence so we could not observe otherwise.]

“All You Ever Wanted to Know about MIRV and ICBM Calculations but Were Not Cleared to Ask”, Davis & Schilling 1973

1973-davis.pdf: “All You Ever Wanted to Know about MIRV and ICBM Calculations but Were Not Cleared to Ask”⁠, Lynn Etheridge Davis, Warner R. Schilling (1973-01-01)

“Can We Survive Technology?”, Neumann 1955

1955-vonneumann-canwesurvivetechnology.pdf: “Can We Survive Technology?”⁠, John von Neumann (1955-01-01)

“Don’t Worry—It Can’t Happen”, Harrington 1940

1940-sciam-harrington-nuclearweapons-dontworryitcanthappen.pdf: “Don’t Worry—It Can’t Happen”⁠, Jean Harrington (1940-05-01; AI scaling; backlinks; similar):

…Early last summer, in the midst of all this research, a chilly sensation began tingling up and down the spines of the experimenters. These extra neutrons that were being erupted—could they not in turn become involuntary bullets, flying from one exploding uranium nucleus into the heart of another, causing another fission which would itself cause still others? Wasn’t there a dangerous possibility that the uranium would at last become explosive? That the samples being bombarded in the laboratories at Columbia University, for example, might blow up the whole of New York City? To make matters more ominous, news of fission research from Germany, plentiful in the early part of 1939, mysteriously and abruptly stopped for some months. Had government censorship been placed on what might be a secret of military importance? The press and populace, getting wind of these possibly lethal goings-on, raised a hue and cry. Nothing daunted, however, the physicists worked on to find out whether or not they would be blown up, and the rest of us along with them. Now, a year after the original discovery, word comes from Paris that we don’t have to worry.

…With typical French—and scientific—caution, they added that this was perhaps true only for the particular conditions of their own experiment, which was carried out on a large mass of uranium under water. But most scientists agreed that it was very likely true in general.

…Readers made insomnious by “newspaper talk” of terrific atomic war weapons held in reserve by dictators may now get sleep.

“Alleged Sins of Science”, Millikan 1930

1930-millikan.pdf: “Alleged Sins of Science”⁠, Robert A. Millikan (1930-02-01)

“Forget about Drones, Forget about Dystopian Sci-fi - a Terrifying New Generation of Autonomous Weapons Is Already Here. Meet the Small Band of Dedicated Optimists Battling Nefarious Governments and Bureaucratic Tedium to Stop the Proliferation of Killer Robots And, Just Maybe, save Humanity from Itself.”

https://www.buzzfeed.com/sarahatopol/how-to-save-mankind-from-the-new-breed-of-killer-robots: “Forget about drones, forget about dystopian sci-fi - a terrifying new generation of autonomous weapons is already here. Meet the small band of dedicated optimists battling nefarious governments and bureaucratic tedium to stop the proliferation of killer robots and, just maybe, save humanity from itself.” (robotics; backlinks)

“Homepage of Paul F. Christiano”, Christiano 2022

“Homepage of Paul F. Christiano”⁠, Paul F. Christiano (GPT⁠, AI scaling⁠, AI safety; backlinks; similar):

I work on AI alignment⁠. I’m a member of the safety team at OpenAI⁠, a research associate at FHI⁠, a board member at Ought⁠, and a recent graduate of the theory group at UC Berkeley.

You may be interested in my writing about alignment⁠, my blog⁠, my academic publications⁠, or my involvement with effective altruism⁠.

You can reach me at paulfchristiano@gmail.com.

Scrupulosity

Wikipedia

“Doing Enough”

2013-evansarieyh-doingenough.html: “Doing Enough” (ethics; backlinks)

“Http://xa.yimg.com/kq/groups/23070378/1331435883/name/Jaan+Tallinn+2011+05+-+revised.doc”

2011-05-10-givewell-holdenkarnofskyjaantallinn.doc: “http: /  / xa.yimg.com / kq / groups / 23070378 / 1331435883 / name / Jaan+Tallinn+2011+05+-+revised.doc” (AI; backlinks)