Overview

2019 was a fun year.

AI: 2019 was a great year for hobbyists and fun generative projects like mine, thanks to spinoffs and especially pretrained models. How much more boring it would have been without the GPT-2 or StyleGAN models! (There was irritatingly little meaningful news about self-driving cars.) More seriously, the theme of 2019 was scaling. Whether GPT-2 or StyleGAN 1+2, or the scaling papers, or AlphaStar, or MuZero, 2019 demonstrated the blessings of scale in scaling up models, compute, data, and tasks; it is no accident that the most extensively discussed editorial on DL/DRL was Rich Sutton’s “The Bitter Lesson”. For all the critics’ carping and goalpost-moving, scaling is working, especially as we go far past the regimes where they assured us years ago that mere size and compute would break down and we would have to use more elegant and intelligent methods like Bayesian program synthesis. Instead, every year it looks increasingly like the strong connectionist thesis is correct: much like humans & evolution, AGI can be reached by training an extremely large number of relatively simple units end-to-end for a long time on a wide variety of multimodal tasks, and it will recursively self-improve meta-learning efficient internal structures & algorithms optimal for the real world which learns how to generalize, reason, self-modify with internal learned reward proxies & optimization algorithms, and do zero/few-shot learning bootstrapped purely from the ultimate reward signals—without requiring extensive hand-engineering, hardwired specialized modules designed to support symbolic reasoning, completely new paradigms of computing hardware etc. (eg. Clune2019). Self-driving cars remain a bitter mystery (although it was nice to see in 2019 Elon Musk & Tesla snatch victory from the jaws of snatching-defeat-from-the-jaws-of-victory).

2019 for genetics saw more progress on genetic-engineering topics than GWASes; the GWASes that did come out were largely confirmatory—no one really needed more SES GWASes from Hill et al, or confirmation that the IQ GWASes work and that brain size is in fact causal for intelligence, and while the recovery of full height/BMI trait heritability from WGS is a strong endorsement of the long-term value of WGS, the transition from limited SNP data to WGS is foreordained (especially since WGS costs appear to finally be dropping again after their long stagnation). Even embryo selection saw greater mainstream acceptance, with a paper in Cell concluding (for the crudest possible simple embryo selection methods) that, fortunately, the glass was half-empty and need not be feared overmuch. (I am pleased to see that as of 2019, human geneticists have managed to reinvent Fisher1918/Lush1943’s breeder’s equation; with any luck, in a few years they may progress as far as reinventing Hazel & Lush1943.) More interesting were the notable events along all axis of post-simple-embryo-selection strategies: Genomic Prediction claimed to have done the first embryo selection on multiple PGSes, genome synthesis saw E. coli achieved, multiple promising post-CRISPR or mass CRISPR editing methods were announced, gene drive progressed to mammals, gametogenesis saw progress (including at least two human fertility startups I know of), serious proposals for human germline CRISPR editing are being made by a Russian (among others), and while He Jiankui was imprisoned by a secret court there otherwise do not appear to have been serious repercussions such as reports of the 3 CRISPR babies being harmed or an ‘indefinite moratorium’ (ie. ban). Thus, we saw good progress towards the enabling technologies for massive embryo selection (breaking the egg bottleneck by allowing generation of hundreds or thousands of embryos and thus multiple-SD gains from selection), IES (Iterated Embryo Selection), massive embryo editing (CRISPR or derivatives), and genome synthesis.

VR’s 2019 launch of Oculus Quest proved quite successful, selling out occasionally well after launch, and appears to appeal to normal people, with even hardcore VR fans acknowledging how much they appreciate the convenience of a single integrated unit. Unfortunately… it is not successful enough. There is no VR wave. Selling out may have as much to do with Facebook not investing too much into manufacturing. Worse, there is still no killer app beyond Beat Saber. The hardware is adequate to the job, the price is nugatory, the experience unparalleled, but there is no stampede into VR. So it seems VR is doomed to the long slow multi-decade adoption slog like that of PCs: it’s too new, too different, and we’re still not sure what to do with it. One day, it would not be surprising if most people have a VR headset, but that day is a long way away.

Bitcoin: little of note. Darknet markets proved unusually interesting: Dream Market, the longest-lived DNM ever, finally expired; Reddit betrayed its users by wholesale purging of subreddits, including /r/DarkNetMarkets, causing me a great deal of grief; and most shockingly, DeepDotWeb was raided by the FBI over affiliate commissions it received from DNMs (apparently into the tens of millions of dollars—you’d’ve thought they’d taken down those hideous ads all over DDW if the affiliate links were so profitable…)

Links

Genetics:

1. Everything Is Heritable:

   1. “Recovery of [full height/BMI] trait heritability from whole genome sequence data”, Wainschtein et al 2019

   2. “The causal influence of brain size on human intelligence: Evidence from within-family phenotypic associations and GWAS modeling”, Lee et al 2019

   3. “Genetic analysis identifies molecular systems and biological pathways associated with household income”, Hill et al 2019a; “Genome-wide analysis identifies molecular systems and 149 genetic loci associated with income”, Hill et al 2019b

   4. “Comparing within. and between-family polygenic score prediction”, Selzam et al 2019

   5. “Large-scale GWAS reveals insights into the genetic architecture of same-sex sexual behavior”, Ganna et al 2019

   6. “Repurposing large health insurance claims data to estimate genetic and environmental contributions in 560 phenotypes”, Lakhani et al 2019

   7. “Extreme inbreeding in a European ancestry sample from the contemporary UK population”, Yengo et al 2019

2. Engineering:

   1. ‘“Embryos have been chosen to reduce disease risk using pre-implantation genetic testing for polygenic traits, and this has resulted in pregnancy,” Laurent Tellier, CEO of Genomic Prediction, told New Scientist.’

   2. full genome synthesis of E. coli: 4 million base-pairs; “Technological challenges and milestones for writing genomes: Synthetic genomics requires improved technologies”, Ostrov et al 2019

   3. “Extensive Mammalian Germline Genome Engineering”, Yang et al 2019; “Enabling large-scale genome editing by reducing DNA nicking”, Smith et al 2019

   4. “Functional Oocytes Derived from Granulosa Cells”, Tian et al 2019; “Controlled modelling of human epiblast and amnion development using stem cells”, Zheng et al 2019

   5. The majority of SK police quarantine dogs are now clones

   6. “Synthetic glycolate metabolism pathways stimulate crop growth and productivity in the field”, South et al 2019

   7. “Super-Mendelian inheritance mediated by CRISPR-Cas9 in the female mouse germline”, Grunwald et al 2019

   8. “Livestock 2.0—genome editing for fitter, healthier, and more productive farmed animals”, Tait-Burkard et al 2018

   9. He Jiankui sentenced to 3 years prison

3. Recent Evolution:

   1. “A genome-wide Approximate Bayesian Computation approach suggests only limited numbers of soft sweeps in humans over the last 100,000 years”, Laval et al 2019

   2. “Highly Heritable and Functionally Relevant Breed Differences in Dog Behavior”, MacLean et al 2019; “Absolute brain size predicts dog breed differences in executive function”, Horschler et al 2019; “Breed differences of heritable behavior traits in cats”, Salonen et al 2019

   3. “Evolutionary Perspectives on Genetic and Environmental Risk Factors for Psychiatric Disorders”, Keller2018

   4. “Thinking About the Evolution of Complex Traits in the Era of Genome-Wide Association Studies”, Sella & Barton2019

   5. “Genetic contributions to variation in human stature in prehistoric Europe”, Cox et al 2019

AI:

1. “The Bitter Lesson”, Sutton2019

2. Language models: “Better Language Models and Their Implications”, OpenAI 2019 (“Language Models are Unsupervised Multitask Learners”, Radford et al 2019)

   • MegatronLM, Nvidia2019

   • “T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, Raffel et al 2019

   • Huggingface

3. “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”, Tan & Le2019

4. “Adversarial Examples Are Not Bugs, They Are Features”, Ilyas et al 2019 (discussion)

5. MuseNet: large Sparse Transformers for MIDI music generation (Child et al 2019)

6. Waifu Labs

7. “Real-time Continuous Transcription with Live Transcribe”

8. reinforcement learning

   1. “MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model”, Schrittwieser et al 2019

   2. “Grandmaster level in StarCraft II using multi-agent reinforcement learning”, Vinyals et al 2019 (blog: [1](https://deepmind.google/discover/blog/alphastar-mastering-the-real-time-strategy-game-starcraft-ii/ “AlphaStar: Mastering the Real-Time Strategy Game StarCraft II”/)/2)

   3. “Dactyl: Solving Rubik’s Cube With A Robot Hand”, Akkaya et al 2019 (blog)

   4. “Dota 2 with Large Scale Deep Reinforcement Learning”, Berner et al 2019 (blog)

   5. ‘Pluribus’: “Superhuman AI for multiplayer poker”, Brown & Sandholm2019

   6. “Meta-learning of Sequential Strategies”, Ortega et al 2019; “Reinforcement Learning, Fast and Slow”, Botvinick et al 2019; “Meta-learners’ learning dynamics are unlike learners’”, Rabinowitz2019; “Ray Interference: a Source of Plateaus in Deep Reinforcement Learning”, Schaul et al 2019

   7. On “Meta Reinforcement Learning”, Lilian Weng

   8. “Learning to Predict Without Looking Ahead: World Models Without Forward Prediction”, Freeman et al 2019

   9. “ICML 2019 Notes”, David Abel

   10. “Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective”, Everitt & Hutter2019

Statistics/meta-science:

1. “Most Rigorous Large-Scale Educational RCTs are Uninformative: Should We Be Concerned?”, Lortie-Forgues & Inglis2019

2. “Meta-Research: A comprehensive review of randomized clinical trials in three medical journals reveals 396 [13%] medical reversals”, Herrera-Perez et al 2019

3. “Comparing meta-analyses and preregistered multiple-laboratory replication projects”, Kvarven et al 2019

4. “The Big Crunch”, David Goodstein1994; “1960: The Year The Singularity Was Cancelled”, Scott Alexander

5. “We Need a New Science of Progress: Humanity needs to get better at knowing how to get better”, Patrick Collison & Tyler Cowen

6. “Ingredients for creating disruptive research teams”, Stefan Torges

7. “Evidence on good forecasting practices from the Good Judgment Project”

8. “The maddening saga of how an Alzheimer’s ‘cabal’ thwarted progress toward a cure for decades”; amyloid does not even predict cognitive decline

9. “Registered reports: an early example and analysis”, Wiseman et al 2019

10. “bioRxiv: the preprint server for biology”, Sever et al 2019

Politics/religion:

1. “Constitutional Hardball”, Tushnet2004

2. “The Real War 1939–1945”, Paul Fussell1989; “Why Men Love War”, Broyles1984

3. Book Review: The Secret Of Our Success

4. “How Gullible are We? A Review of the Evidence from Psychology and Social Science”, Mercier2017

5. How Soviet centralized economic planning led to the near-extinction of whales, for no particular reason

6. “Avoid News: Towards a Healthy News Diet”, Dobelli2010

7. “Aztec Political Thought”; “Nggwal, who travels in structures of fiber and bone atop rivers of blood.”

8. “Mau-Mauing the Flak Catchers”, Tom Wolfe 1970

9. “Anthropology’s Science Wars: Insights from a New Survey”, Horowitz et al 2019 (visualizations)

10. “Dying the Christian Science way: the horror of my father’s last days; The anti-medical dogma of Christian Science led my father to an agonising death. Now the church itself is in decline—and it can’t happen fast enough”

Psychology:

1. “A century of research on conscientiousness at work”, Wilmot & Ones2019

2. “How Replicable Are Links Between Personality Traits and Consequential Life Outcomes? The Life Outcomes of Personality Replication Project”, Soto2019

3. “The impact of background music on adult listeners: A meta-analysis”, Kampfe et al 2011

4. “On Bloom’s two sigma problem: A systematic review of the effectiveness of mastery learning, tutoring, and direct instruction”

5. “Stereotype Threat Effects in Settings With Features Likely Versus Unlikely in Operational Test Settings: A Meta-Analysis”, Shewach et al 2019

6. “Debunking the Stanford Prison Experiment”, Le Texier2019; Did psychologist David Rosenhan fabricate his famous 1973 “Being Sane in Insane Places” mental hospital exposé?; “A real-life Lord of the Flies: the troubling legacy of the Robbers Cave experiment”

7. “What’s Wrong with Psychology Anyway?”, Lykken1991

8. “What the voice inside your head says about you: We tend to assume that our internal monologue ‘speaks’ in words—but it turns out that, for many of us, it’s much more complicated” (see also “Toward a phenomenology of inner speaking”, Hurlburt et al 2013)

9. “Doing Despite Disliking: Self-regulatory Strategies in Everyday Aversive Activities”, Hennecke et al 2018 (see also Structured Procrastination)

10. “Enjoy It Again: Repeat Experiences Are Less Repetitive Than People Think”, O’Brien2019

Biology:

1. “Restoration of brain circulation and cellular functions hours post-mortem”, Vrselja et al 2019

2. “A Look Back at 2019: Progress Towards the Treatment of Aging as a Medical Condition”, Reason

3. “Widespread associations between grey matter structure and the human phenome”, Couvy-Duchesne et al 2019

4. “Deep Intellect”: the mind of an octopus

5. “Invisible Designers: Brain Evolution Through the Lens of Parasite Manipulation”, Del Giudice2019

6. “The Family That Feels Almost No Pain: An Italian clan’s curious insensitivity to pain has piqued the interest of geneticists seeking a new understanding of how to treat physical suffering”

7. “Congregations Of Tear-Drinking Bees At Human Eyes: Foraging Strategies For An Invaluable Resource By Lisotrigona In Thailand (Apidae, Meliponini)”, Bänziger2018

Technology:

1. “STEPS Toward Expressive Programming Systems: ‘A Science Experiment’”, Ohshima et al 2012

2. “How can we develop transformative tools for thought?”, Andy Matuschak & Michael Nielsen 2019

3. “How Complex Systems Fail”, Cook2000

4. “The Cosmic Distance Ladder”, Terence Tao 2010

5. “Why did we wait so long for the bicycle?”

6. “Hall’s Law: The Nineteenth Century Prequel to Moore’s Law”

7. “Measuring Long-term Impact of Ads on LinkedIn Feed”, Yan et al 2019

8. “href.cool: Links of the 2010s”, Kicks Condor

9. “Chain Letter Evolution”

10. “Mother Earth Mother Board”, Neal Stephenson 1996

Economics:

1. “Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure”, Nadia Eghbal2016

2. “The changing structure of American innovation: Some cautionary remarks for economic growth”, Arora et al 2020

3. “End-To-End Arguments In System Design”, Saltzer et al 1984; “Observations on Errors, Corrections, & Trust of Dependent Systems”, James Hamilton (the end-to-end principle)

4. “Anthropological invariants in travel behavior”, Marchetti1994

5. “Labour repression—the Indo-Japanese divergence”, Pseudoerasmus

6. “The Optimistic Thought Experiment”, Thiel 2008

7. “The Great Shift in Japanese Pop Culture—Part One”/Part Two/Part Three/Part Four/Part Five

8. “Managing an iconic old luxury brand in a new luxury economy: Hermès handbags in the US market”, Lewis & Haas2014 (summary; see also “Birkin Demand: A Sage & Stylish Investment”, Newsom2016)

9. “The Price of Nails since 1700: Even Simple Products Experienced Large Price Declines”, Sichel2017

10. “The Corporate Governance of Benedictine Abbeys: What can Stock Corporations Learn from Monasteries?”, Rost et al 2010

Philosophy:

1. “What Color are your bits?”

2. “Two Arms and a Head: The Death of a Newly Paraplegic Philosopher”, Atreus2008

3. “Possible Girls”, Sinhababu2008

4. “The Force That Drives the Flower”, Dillard1973

5. “Consider the Lobster: For 56 years, the Maine Lobster Festival has been drawing crowds with the promise of sun, fun, and fine food. One visitor would argue that the celebration involves a whole lot more”, David Foster Wallace 2004

Fiction:

1. RIP Gene Wolfe

2. “A Colder War”, Charles Stross2000

3. “The Proverbial Murder Mystery”, SSC, SSC (see also “The Study of Anglophysics”)

4. “A science fiction writer of the Fifties”, Brad Leithauser 2006

5. “Killing Rabbits”, Miroslav Válek1963

6. “The First Sally (A), or, Trurl’s Electronic Bard”, Stanisław Lem

7. “How a Literary Prank Convinced Germany That ‘Hansel and Gretel’ Was Real: A 1963 book purported to prove that the siblings were murderous bakers”

8. “Neon Genesis Evangelion: Graphic designer Peiran Tan plumbs the typographic psyche of the celebrated anime franchise”; Typography in Alien & Blade Runner

9. “The Inner Life of Chinese Teenagers”

10. Roar (1981 film)

Books

Nonfiction:

1. The Cultural Revolution: A People’s History, 1962–1976, Dikötter 2016 (review)

2. Visual Explanations: Images and Quantities, Evidence and Narrative, Tufte 1997 (review)

TV/movies

Nonfiction movies:

1. Free Solo (review)

2. Weiner (review)

Fiction:

1. Carmen (review)

2. Akhnaten (review)

3. Stalker (review)

4. Sicario

5. Freaks, 1932 (review)

6. Manon (review)

7. Die Walküre (review)

8. Madama Butterfly (review)

9. Invasion of the Body Snatchers, 1978 (review)

10. Rurouni Kenshin 2012/Rurouni Kenshin: Kyoto Inferno 2014/Rurouni Kenshin: The Legend Ends 2014 (review)

Anime: