--- title: September 2020 News description: September 2020 Gwern.net newsletter with links on DRL and AI scaling, psychiatric disorders; no reviews. created: 2019-12-26 status: finished previous: /newsletter/2020/08 next: /newsletter/2020/10 confidence: log cssExtension: dropcaps-de-zs backlink: False ... September 2020's [Gwern.net](/newsletter/2020/09 "'September 2020 News', Branwen 2019") [newsletter](https://gwern.substack.com/ "'Gwern.net newsletter (Substack subscription page)', Branwen 2013") is now out; previous, [August 2020](/newsletter/2020/08 "'August 2020 News', Branwen 2019") ([archives](/doc/newsletter/index)). This is a collation of links and summary of major changes, overlapping with my [Changelog](/changelog); brought to you by my donors on [Patreon](https://www.patreon.com/gwern). # Writings - **Gwern.net**: dark mode image handling fixed (images are inverted based on automated heuristic + manual tagging); annotations can now be 'definitions' (popup annotations for non-link text); expanded tooltips as fallback for link annotations; bolded abstracts & revised list hierarchy to use bold; changed all dates to YYYY-MM-DD for consistency; added missing link icons for >7 domains, fixed spacing & overlapping, and harmonized opacity of all link icons; Pandoc now uses MathJax directly, enabling colored equations; internal links to a previous or later section now point up or down (respectively) as a navigation aid; experimental use of [instant.page](https://instant.page/ "'Instant.page: Make your site’s pages instant in 1 minute and improve your conversion rate noticeably.', Dieulot 2019") prefetching; [custom](/static/img/ornament/asterism-triplewhitestar.svg) [asterism](https://en.wikipedia.org/wiki/Asterism_\(typography\)) instead of horizontal ruler; simplified mobile appearance; miscellaneous bug fixes (assisted by new [test/demo page](/lorem "'Lorem Ipsum', Branwen 2020")) # Links ## AI - ["ReBeL: Combining Deep Reinforcement Learning and Search for Imperfect-Information Games"](https://arxiv.org/abs/2007.13544#facebook){#rebel}, Brown et al 2020 (heads-up no-limit Texas hold'em poker close to solved with a generalizable AlphaZero-like approach?) - ["Assessing Game Balance with AlphaZero: Exploring Alternative Rule Sets in Chess"](https://arxiv.org/abs/2009.04374#deepmind), Tomašev et al 2020 - ["A digital biomarker of diabetes from smartphone-based vascular signals"](/doc/ai/nn/cnn/2020-avram.pdf), Avram et al 2020 (time-series of blood pulses can diagnose diabetes? weird, but not as weird as [predicting gender from retinas](/doc/ai/nn/2018-poplin.pdf "'Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning', Poplin et al 2018")) - ["Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves"](https://arxiv.org/abs/2009.11243#google), Metz et al 2020; ["Gradient Descent: The Ultimate Optimizer"](https://arxiv.org/abs/1909.13371#facebook), Chandra et al 2019 (deeper---we need to go deeper... now [scales to VGG-Net](https://twitter.com/headinthebox/status/1224098260140224512 "Reviewer #2 doubted that our hyper-hyperparameter optimizer (https://arxiv.org/abs/1909.13371) would work for larger examples. Of course it does, but thanks for asking. (Erik Meijer)")) - ["X-LXMERT/X-UNITER: Paint, Caption and Answer Questions with Multi-Modal Transformers"](https://arxiv.org/abs/2009.11278#allen "'X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers', Cho et al 2020"), Cho et al 2020 ([online demo](https://vision-explorer.allenai.org/text_to_image_generation "Text-to-Image Generation: Text-to-Image Generation is the task of generating an image conditioned on the input text. Try it for yourself: 1. Choose a Caption"); finally, some new work on text → image) - ["StyleGAN network blending"](https://www.justinpinkney.com/blog/2020/stylegan-network-blending/), Justin Pinkney/["Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains"](https://arxiv.org/abs/2010.05334 "‘Toonify: Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains’, Pinkney & Adler 2020"), Pinkney & Adler 2020 (interpolating between models to transfer images between domains, eg. FFHQ ↔ foxes ↔ anime ↔ MLP ↔ furries) - ["Optimal Peanut Butter and Banana Sandwiches"](https://www.ethanrosenthal.com/2020/08/25/optimal-peanut-butter-and-banana-sandwiches/), Ethan Rosenthal ([bin packing](https://en.wikipedia.org/wiki/Bin_packing_problem) banana slices) Matters Of Scale: - ["DeepSpeed: Extreme-scale model training for everyone"](https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/) (demonstrates training of GPT-3-180b & 1t-parameter models ("The trillion-parameter model has 298 layers of Transformers with a hidden dimension of 17,408 and is trained with sequence length 2,048 and batch size 2,048"), w/open-source code; able to use CPU+GPU RAM simultaneously for 13b-parameter models on each node per [Pudipeddi et al 2020](https://arxiv.org/abs/2002.05645#microsoft "Training Large Neural Networks with Constant Memory using a New Execution Algorithm"); [sparse attention](https://www.deepspeed.ai/news/2020/09/08/sparse-attention.html "DeepSpeed Sparse Attention") for saving RAM; approximated Adam gradients for saving bandwidth) - ["GPT-f: Generative Language Modeling for Automated Theorem Proving"](https://arxiv.org/abs/2009.03393#openai "'Generative Language Modeling for Automated Theorem Proving', Polu & Sutskever 2020"), Polu & Sutskever 2020b (GPT-2 for [Metamath](https://en.wikipedia.org/wiki/Metamath) scales & can bootstrap its theorem-proving ability---[onward to IMO!](https://twitter.com/spolu/status/1303693595833237504 "The examples are indeed extremely simple on purpose (otherwise it's hard to communicate efficiently what's happening to non Metamath experts). That being said, we're still pretty far away from IMOs; but this is definitely a goal for us, and one we're actively working towards!")) - ["Measuring Massive Multitask Language Understanding"](https://arxiv.org/abs/2009.03300 "‘MMLU: Measuring Massive Multitask Language Understanding’, Hendrycks et al 2020"), Hendrycks et al 2020 (introducing new much harder benchmarks to stress-test models---nevertheless, bigger=better: [GPT-3 model size vs Q&A](/doc/ai/nn/transformer/gpt/2020-hendrycks-figure1b-gpt3-qascaling.png "Figure 1b: GPT-3 Few Shot Test Performance: Performance on a commonsense benchmark (HellaSwag), a linguistic understanding benchmark (Super-GLUE), and the massive multitask test. On previous benchmarks, smaller models start well above random chance levels and exhibit more continuous improvements with model size increases, but on our test, GPT-3 moves beyond random chance with the largest model"){.invert}, with GPT-3-173b hitting almost 50% out of 4-choice questions vs baseline models like T5 at the random baseline of only 25%) - ["DynamicEmbedding: Extending TensorFlow for Colossal-Scale Applications"](https://arxiv.org/abs/2004.08366#google), Zeng et al 2020 (embeddings-as-a-service Google-wide; [tool AIs want to be agent AIs](/tool-ai "'Why Tool AIs Want to Be Agent AIs', Branwen 2016")...) - ["A Time Leap Challenge for SAT Solving"](https://arxiv.org/abs/2008.02215), Fichte et al 2020 ([your computer is faster than you think](/note/faster "'Computer Optimization: Your Computer Is Faster Than You Think', Branwen 2021"), software+hardware overhang: ~2.5× performance increase in [SAT solving](https://en.wikipedia.org/wiki/Boolean_satisfiability_problem#Algorithms_for_solving_SAT) since 2000, about equally due to software & hardware gains, although slightly more software---new software on old hardware beats old on new. See previously [Grace 2013](/doc/ai/scaling/2013-grace.pdf#miri "Algorithmic Progress in Six Domains").) - ["Efficient Transformers: A Survey"](https://arxiv.org/abs/2009.06732#google), Tay et al 2020 (more on [efficient attention](/note/attention "'Efficient Attention: Breaking The Quadratic Transformer Bottleneck', Branwen 2020") approaches) ## Genetics Everything Is Heritable: - ["A general dimension of genetic sharing across diverse cognitive traits inferred from molecular data"](/doc/iq/2020-delafuente.pdf), de la Fuente et al 2020 - ["Mapping genomic loci prioritises genes and implicates synaptic biology in schizophrenia"](https://www.medrxiv.org/content/10.1101/2020.09.12.20192922.full), PGC et al 2020 (PGC SCZ3: [248 genome-wide hits](/doc/genetics/heritable/correlation/2020-pgc-figure2-gwasprogressovertime.png "Extended Data Figure 2: GWAS progress over time. The relationship of GWAS associations to sample-size is shown in this plot with selected SCZ GWAS meta-analyses of the past 11 years. The x-axis shows number of cases. The y-axis shows the number of independent loci discovered with at least one genome-wide statistically-significant index SNP in the discovery meta-analysis (eg. without replication data)...The slope of ~4 newly discovered loci per 1000 cases between 2013 and 2019 increased to a slope of ~6 with the latest sample-size increase."){.invert}; 7.7% PGS; OR = 44 comparing top vs bottom percentile risk); ["Exome sequencing identifies rare coding variants in 10 genes which confer substantial risk for schizophrenia"](https://www.medrxiv.org/content/10.1101/2020.09.18.20192815.full), Singh et al 2020 ([figure 6a](/doc/genetics/heritable/rare/2020-singh-figure6a-thecontributionsofultrarareptvstoschizophreniarisk.png "Figure 6. The contributions of ultra-rare PTVs [protein-truncating variants] to schizophrenia risk. A: Genetic architecture of schizophrenia. Significant genetic associations for schizophrenia from the most recent GWAS, CNV, and sequencing studies are displayed. The in-sample odds ratio is plotted against the minor allele frequency in the general population. The color of each dot corresponds to the source of the association, and the size of the dot to the odds ratio. The shaded area represented the LOESS-smoothed lines of the upper and lower bounds of the point estimates...Because schizophrenia as a trait is under strong selection^38–40^, we expect that URVs of large effect to be frequently de novo or of very recent origin and contribute to risk in only a fraction of diagnosed patients.") is a striking demonstration of [negative/purifying selection](https://en.wikipedia.org/wiki/Negative_selection_\(natural_selection\)): more harmful mutations are rarer) - ["Genome-wide association study of over 40,000 bipolar disorder cases provides novel biological insights"](https://www.medrxiv.org/content/10.1101/2020.09.17.20187054.full), Mullins et al 2020 (PGC3 BPD; PGS: 4.75%) - ["Genetic Fortune: Winning or Losing Education, Income, and Health"](https://papers.tinbergen.nl/20053.pdf#page=4), Kweon et al 2020 (within-family education/income/health PGSes) - ["GWAS of Over 427,000 Individuals Establishes GABAergic and Synaptic Molecular Pathways as Key for Cognitive Executive Functions"](https://www.biorxiv.org/content/10.1101/674515.full), Hatoum et al 2020 - ["Shared heritability of face and brain shape distinct from cognitive traits"](https://www.biorxiv.org/content/10.1101/2020.08.29.269258.full), Naqvi et al 2020 (setting limits to facial prediction of some phenotypes) Recent Evolution: - ["Rapid Evolution of Plastic-degrading Enzymes Prevalent in the Global Ocean"](https://www.biorxiv.org/content/10.1101/2020.09.07.285692.full), Alam et al 2020 Engineering: - The [first successful cloning](https://reviverestore.org/projects/przewalskis-horse/ "'The Przewalski’s Horse Project', Restore 2020") of the endangered [Przewalski’s horse](https://en.wikipedia.org/wiki/Przewalski%27s_horse) - ["Human Embryo Gene Editing Gets a Road Map—Not a Green Light"](https://www.wired.com/story/human-embryo-gene-editing-gets-a-road-map-not-a-green-light/ "'Human Embryo Gene Editing Gets a Road Map—Not a Green Light: After the 2018 “Crispr baby” scandal, a global commission assessed the technology and set strict criteria for moving it toward clinical trials', Molteni 2020") (National Academy of Science's [_Heritable Human Genome Editing_](https://nap.nationalacademies.org/read/25665/chapter/1) 2020 report) - ["CRISPR-engineered human brown-like adipocytes prevent diet-induced obesity and ameliorate metabolic syndrome in mice"](/doc/genetics/editing/2020-wang-4.pdf), Wang et al 2020; ["CRISPR-enhanced human adipocyte 'browning' as cell therapy for metabolic disease"](https://www.biorxiv.org/content/10.1101/2020.10.13.337923.full), Tsagkaraki et al 2020 ## Statistics/Meta-Science - ["Blueberry Earth"](https://arxiv.org/abs/1807.10553), [Sandberg](https://en.wikipedia.org/wiki/Anders_Sandberg) 2018 ## Politics/Religion - ["America in Decay: The Sources of Political Dysfunction"](/doc/history/2014-fukuyama.pdf), [Fukuyama](https://en.wikipedia.org/wiki/Francis_Fukuyama) 2014 (see also: [constitutional hardball](/doc/sociology/2004-tushnet.pdf "'Constitutional Hardball', Tushnet 2004")/["The Empty Chamber"](https://www.newyorker.com/magazine/2010/08/09/the-empty-chamber "'The Empty Chamber: Just how broken is the Senate?', Packer 2010")) - ["'Dwarf Pride' Was Hard Won. Will a Growth Drug Undermine It? An experimental medication that increases height in children with the most common form of dwarfism has raised hope that it can help them lead easier lives. But some say the condition is not a problem in need of a cure."](https://www.nytimes.com/2020/09/05/world/dwarfism-vosoritide.html), [FDA-approved](https://www.statnews.com/2021/11/19/fda-approves-first-drug-for-genetic-cause-of-dwarfism/ "FDA approves first drug for genetic cause of dwarfism") ([Savarirayan et al 2020](/doc/science/2020-savarirayan.pdf "Once-daily, subcutaneous vosoritide therapy in children with achondroplasia: a randomized, double-blind, phase 3, placebo-controlled, multicentre trial"); amusing trivia: the reason why crossing any dog breed with [Corgis](https://en.wikipedia.org/wiki/Welsh_Corgi) gives you a Corgified version of that dog breed is because Corgis are [homozygous dominant](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2748762/ "'An Expressed Fgf4 Retrogene Is Associated with Breed-Defining Chondrodysplasia in Domestic Dogs', Parker et al 2009") on an [FGF4](https://en.wikipedia.org/wiki/FGF4) mutation causing similar [achondroplasia](https://en.wikipedia.org/wiki/Achondroplasia) as in humans) - ["Laws of Xmas"](http://linuxmafia.com/~rick/xmas.html "'Laws of Xmas [Have you ever wondered what Xmas would be like if it were a Jewish Holiday?…]', Miller & Miller 1998"), Akiva & Ilene Miller 1998 - [Gruinard Island](https://en.wikipedia.org/wiki/Gruinard_Island) - [London Necropolis railway](https://en.wikipedia.org/wiki/London_Necropolis_Railway) [station](https://en.wikipedia.org/wiki/London_Necropolis_railway_station) ## Psychology/Biology - ["Self-Supervised Natural Image Reconstruction and Rich Semantic Classification from Brain Activity"](https://www.biorxiv.org/content/10.1101/2020.09.06.284794.full), Gaziv et al 2020 (towards [brain imitation learning](https://www.reddit.com/r/reinforcementlearning/comments/9pwy2f/wbe_and_drl_a_middle_way_of_imitation_learning/ "'WBE and DRL: a Middle Way of imitation learning from the human brain', Branwen 2018")) - ["Catching Cowpox: The Early Spread of Smallpox Vaccination, 1798–1810"](/doc/biology/2009-rusnock.pdf), Rusnock 2009 - ["Creativity and Intelligence: An Investigation of the Threshold Hypothesis"](https://osf.io/preprints/psyarxiv/89bv5/), Weiss et al 2020 - [Discovery of more water on Mars](https://www.nature.com/articles/d41586-020-02751-1 "Water on Mars: discovery of three buried lakes intrigues scientists: Researchers say they have detected a group of lakes hidden under the red planet’s icy surface.") and [first extra-galactic planet](https://arxiv.org/abs/2009.08987 "'M51-ULS-1b: The First Candidate for a Planet in an External Galaxy', Di Stefano et al 2020") - [The Collyer Brothers](https://en.wikipedia.org/wiki/Collyer_brothers); [The Yuba County Five](https://en.wikipedia.org/wiki/Yuba_County_Five) ## Technology - ["Unraveling The JPEG"](https://parametric.press/issue-01/unraveling-the-jpeg/ "'Unraveling the JPEG: JPEG images are everywhere in our digital lives, but behind the veil of familiarity lie algorithms that remove details that are imperceptible to the human eye. This produces the highest visual quality with the smallest file size—but what does that look like? Let's see what our eyes can't see!', Shehata 2019"), Omar Shehata 2019 (explorable: editing JPEGs to understand [their encoding](https://en.wikipedia.org/wiki/JPEG#Quantization)) - ["The Recursive Universe"](https://blog.amandaghassaei.com/2020/05/01/the-recursive-universe/), Amanda Ghassaei (how to implement [Conway's](https://en.wikipedia.org/wiki/John_Horton_Conway) [Game of Life](https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life) inside itself as a single giant ['metapixel'](https://conwaylife.com/wiki/OTCA_metapixel) pattern) - ["The First Roman Fonts"](https://ilovetypography.com/2016/04/18/the-first-roman-fonts/), John Boardley - [Naumachia](https://en.wikipedia.org/wiki/Naumachia) ## Economics - ["Salary Negotiation: Make More Money, Be More Valued"](https://www.kalzumeus.com/2012/01/23/salary-negotiation/), patio11 - ["A Deep Dive into K-pop"](https://mattlakeman.org/2020/09/06/a-deep-dive-into-k-pop/), Matt Lakeman - ["Wikipedia Matters"](https://www.marit.hinnosaar.net/wikipediamatters.pdf), Hinnosaar et al 2019 (randomization of edits to foreign languages' Wikipedia entries about an obscure Spanish city increases tourism to the city from those foreign countries by 9%) ## Miscellaneous - [1955 Le Mans disaster](https://en.wikipedia.org/wiki/1955_Le_Mans_disaster) (the past is a Third World country)