AI

• “Human-Level Per­for­mance in No-Press Diplo­macy via Equi­lib­rium Search”, Gray et al 2020
• “Dream­erV2: Mas­ter­ing Atari with Dis­crete World Mod­els”, Hafner et al 2020
• “Tax­on­omy of Real Faults in Deep Learn­ing Sys­tems”, Hum­bat­ova et al 2019 (on the painfully prim­i­tive & error-prone tool­ing of AI research (such a con­trast to Lisp machines, where the tool­ing was supe­rior to today’s DL ecosys­tem but the AI research was use­less)—“neural net­works want to work”, so how many seri­ous bugs do we never real­ize? Every researcher has a sto­ry, like R2D2, of how their code had a fatal bug but worked any­way… Just not nearly as well as it should have. Recently in Ten­sor­fork we learned our BigGAN sam­ples did­n’t work—be­cause the Batch Norm sta­tis­tics weren’t being used at run­time like we thought!)
• “Arti­fi­cial Intel­li­gence ver­sus Maya Angelou: Exper­i­men­tal evi­dence that peo­ple can­not differ­en­ti­ate AI-gen­er­ated from human-writ­ten poetry”, Köbis & Mossink 2020 (hu­mans pre­ferred screened GPT-2 poem com­ple­tions 43% of the time)
• “Ten­sor­flow Neural Net­work Play­ground” (ex­plorable: train small NNs in-browser to watch how their deci­sion bound­aries evolve)

Mat­ters Of Scale:

• “Scal­ing Laws for Autore­gres­sive Gen­er­a­tive Mod­el­ing”, Henighan et al 2020 (Kaplan talk; GPT-3 was not a fluke nor lan­guage-speci­fic: all modal­i­ties test­ed—­math, video, images, text, com­bined—s­cale cleanly and in the same way where big­ger mod­els = bet­ter; the unsupervised/pretrained mod­els then trans­fer to super­vised learn­ing, like image clas­si­fi­ca­tion. GPT-3 all the things!)
• “Vision Trans­former (ViT): An Image is Worth 16×16 Words: Trans­form­ers for Image Recog­ni­tion at Scale”, Anony­mous et al 2020 (as expect­ed, pure Trans­form­ers sur­pass CNN SOTA when scaled appro­pri­ate­ly: as fast, bet­ter, sim­pler with fewer para­me­ters and cheaper to train, with improve­ments to come: “Final­ly, [we plan] to fur­ther scale ViT, given that the per­for­mance does not seem yet to be sat­u­rat­ing with the increased model size…” lucidrain­s’s imple­men­ta­tion)
• “Meta-trained agents imple­ment Bayes-op­ti­mal agents”, Miku­lik et al 2020; “One Epoch Is All You Need”, Komat­suzaki 2019 (are big­ger deep mod­els bet­ter because they meta-learn Bayes-op­ti­mal infer­ence?)
• NMT: “mT5: A mas­sively mul­ti­lin­gual pre-trained tex­t-to-text trans­former”, Xue et al 2020 (13b-pa­ra­me­ter­s); “M2M-100: Beyond Eng­lish-Cen­tric Mul­ti­lin­gual Machine Trans­la­tion”, Fan et al 2020 (blog; 15b-pa­ra­me­ters, using ZeRO)
• “Push­ing the Lim­its of Semi­-Su­per­vised Learn­ing for Auto­matic Speech Recog­ni­tion”, Zhang et al 2020 (pre­train­ing enables scal­ing to 1b-pa­ra­me­ter effi­cien­t-at­ten­tion Con­form­ers)

Genetics:

Every­thing Is Her­i­ta­ble:

• “Mul­ti­vari­ate genomic analy­sis of 1.5 mil­lion peo­ple iden­ti­fies genes related to addic­tion, anti­so­cial behav­ior, and health”, Lin­ner et al 2020; “Hap­pi­ness and Well­be­ing; the value and find­ings from genetic stud­ies”, Van de Wei­jer et al 2020
• “Com­bined Util­ity of 25 Dis­ease and Risk Fac­tor Poly­genic Risk Scores for Strat­i­fy­ing Risk of All-Cause Mor­tal­ity”, Meis­ner et al 2020
• “Exome sequenc­ing and char­ac­ter­i­za­tion of 49,960 indi­vid­u­als in the UK Biobank”, Van Hout et al 2020 (“2% of this pop­u­la­tion has a med­ically action­able vari­ant”)
• “Explor­ing the vari­ance in com­plex traits cap­tured by DNA methy­la­tion assays”, Bat­tram et al 2020 (epigenome heritability/variance-components: ~0 across 400 traits)

Engi­neer­ing:

• “The sec­ond decade of syn­thetic biol­o­gy: 2010–2020”, Meng & Ellis 2020
• Emmanuelle Char­p­en­tier & Jen­nifer Doudna win Nobel Prize for CRISPR

Statistics/Meta-Science/Math

• “Pub­li­ca­tion rate in pre­clin­i­cal research: a plea for pre­reg­is­tra­tion”, van der Naald et al 2020 (media; ~75% of sac­ri­ficed ani­mals yield no pub­lished result)
• “The Empir­i­cal Meta­math­e­mat­ics of Euclid and Beyond”, Stephen Wol­fram (an­a­lyz­ing Ele­ments as a DAG); “Singing Euclid: the oral char­ac­ter of Greek geom­e­try”, Vik­tor Blåsjö (see also “Hav­ing Had No Pre­de­ces­sor to Imi­tate, He Had No Suc­ces­sor Capa­ble of Imi­tat­ing Him”)

Politics/Religion

• Imma­nen­tiz­ing the Escha­ton, Anabap­tist edi­tion: Mün­ster rebel­lion; the wreck of the Batavia

Psychology/Biology

• “Effects of caloric restric­tion on human phys­i­o­log­i­cal, psy­cho­log­i­cal, and behav­ioral out­comes: high­lights from CALERIE phase 2”, Dor­ling et al 2020
• “Ques­tions About Trees”, Brian Hayes (“the para­dox of the plank­ton”: what main­tains diver­sity in forest­s?)
• “The hear­ing aid dilem­ma: ampli­fi­ca­tion, com­pres­sion, and dis­tor­tion of the neural code”, Arm­strong et al 2020 (to­wards hear­ing aids designed by black­box opti­miza­tion to make neural pat­terns match pat­terns of nor­mal hear­ing?)
• “The role of cat eye nar­row­ing move­ments in cat-hu­man com­mu­ni­ca­tion”, Humphrey et al 2020
• Encephali­tis lethar­gica (“A viral infec­tion of the mind? The curi­ous case of encephali­tis lethar­gica”)
• “Brain Train­ing Habits Are Not Asso­ci­ated With Gen­er­al­ized Ben­e­fits to Cog­ni­tion: An Online Study of Over 1000 ‘Brain Train­ers’”, Sto­janoski et al 2020
• “C₆₀ in olive oil causes light–de­pen­dent tox­i­c­ity and does not extend lifes­pan in mice”, Grohn et al 2020 (Back in 2012, I found the orig­i­nal result unbe­liev­able for many rea­sons¹; but in case any­one still takes it seri­ous­ly…)

Technology

• LibGen/Sci-Hub full­text search engine: search 5.5m+ down­load­able books/77.5m+ papers (great alter­na­tive to Google Books/Scholar)
• “Pro­fil­ing a ware­house­-s­cale com­puter”, Kanev et al 2015 (pre­vi­ous­ly: Ren et al 2010)
• “Expert judg­ment on mark­ers to deter inad­ver­tent human intru­sion into the Waste Iso­la­tion Pilot Plant”, Trauth et al 1993 (the San­dia long-term nuclear waste warn­ing mes­sage expert report for the WIP)
• “The Elu­sive Peril of Space Junk: Mil­lions of human arti­facts cir­cle the Earth. Can we clean them up before they cause a dis­as­ter?” (the Kessler syn­drome never went away)
• “The rise and fall of Adobe Flash: Before Flash Player sun­sets this Decem­ber, we talk its legacy with those who built it”; “How Flash Games Shaped The Video Game Indus­try: Flash is dead. But the influ­ence of Flash games on mod­ern game­play is inescapable”, Jonas Rich­ner (why were Flash browser games so cre­ative? The Flash scene ben­e­fited from easy entry using an inte­grated pro­gram­ming envi­ron­ment, smal­l­-s­cale games enabling rapid iter­a­tion, many exem­plars to imi­tate, envy and admi­ra­tion dri­ving imi­ta­tors, the con­vic­tion that improve­ments were pos­si­ble, a global tal­ent pool with lots of free time, and game web­sites like New­grounds to curate games to cre­ate a sophis­ti­cated & demand­ing user­base which could drive mega-traffic to hits, in a vir­tu­ous cycle of recruitment/competition/feedback/improvement. See also Bakewell.)
• “This page is a truly naked, Bru­tal­ist HTML quine.”; A self­-build­ing web­site

Fiction

• “My Dad, the Pornog­ra­pher” (on Andrew J. Offutt)
• The Mon­kees (from pass­ing ref­er­ences, I was always con­fused if ‘The Mon­kees’ were a TV show or band—ap­par­ently they were both! Best-selling ones, too.)