AI

• “Hu­man-Level Per­for­mance in No-Press Diplo­macy via Equi­lib­rium Search”, Gray et al 2020
• “Dream­erV2: Mas­ter­ing Atari with Dis­crete World Mod­els”, Hafner et al 2020
• “Tax­on­omy of Real Faults in Deep Learn­ing Sys­tems”, Hum­bat­ova et al 2019 (on the painfully prim­i­tive & er­ror-prone tool­ing of AI re­search (such a con­trast to Lisp ma­chines, where the tool­ing was su­pe­rior to to­day’s DL ecosys­tem but the AI re­search was use­less)—“neural net­works want to work”, so how many se­ri­ous bugs do we never re­al­ize? Every re­searcher has a sto­ry, like R2D2, of how their code had a fa­tal bug but worked any­way… Just not nearly as well as it should have. Re­cently in Ten­sor­fork we learned our BigGAN sam­ples did­n’t work—be­cause the Batch Norm sta­tis­tics weren’t be­ing used at run­time like we thought!)
• “Ar­ti­fi­cial In­tel­li­gence ver­sus Maya An­gelou: Ex­per­i­men­tal ev­i­dence that peo­ple can­not differ­en­ti­ate AI-gen­er­ated from hu­man-writ­ten po­etry”, Köbis & Mossink 2020 (hu­mans pre­ferred screened GPT-2 poem com­ple­tions 43% of the time)
• “Ten­sor­flow Neural Net­work Play­ground” (ex­plorable: train small NNs in­-browser to watch how their de­ci­sion bound­aries evolve)

Mat­ters Of Scale:

• “Scal­ing Laws for Au­tore­gres­sive Gen­er­a­tive Mod­el­ing”, Henighan et al 2020 (Ka­plan talk; GPT-3 was not a fluke nor lan­guage-speci­fic: all modal­i­ties test­ed—­math, video, im­ages, text, com­bined—s­cale cleanly and in the same way where big­ger mod­els = bet­ter; the un­su­per­vised/pre­trained mod­els then trans­fer to su­per­vised learn­ing, like im­age clas­si­fi­ca­tion. GPT-3 all the things!)
• “Vi­sion Trans­former (ViT): An Im­age is Worth 16×16 Words: Trans­form­ers for Im­age Recog­ni­tion at Scale”, Anony­mous et al 2020 (as ex­pect­ed, pure Trans­form­ers sur­pass CNN SOTA when scaled ap­pro­pri­ate­ly: as fast, bet­ter, sim­pler with fewer pa­ra­me­ters and cheaper to train, with im­prove­ments to come: “Fi­nal­ly, [we plan] to fur­ther scale ViT, given that the per­for­mance does not seem yet to be sat­u­rat­ing with the in­creased model size…” lu­cidrain­s’s im­ple­men­ta­tion)
• “Meta-trained agents im­ple­ment Bayes-op­ti­mal agents”, Miku­lik et al 2020; “One Epoch Is All You Need”, Ko­mat­suzaki 2019 (are big­ger deep mod­els bet­ter be­cause they meta-learn Bayes-op­ti­mal in­fer­ence?)
• NMT: “mT5: A mas­sively mul­ti­lin­gual pre-trained tex­t-to-text trans­former”, Xue et al 2020 (13b-pa­ra­me­ter­s); “M2M-100: Be­yond Eng­lish-Cen­tric Mul­ti­lin­gual Ma­chine Trans­la­tion”, Fan et al 2020 (blog; 15b-pa­ra­me­ters, us­ing Ze­RO)
• “Push­ing the Lim­its of Semi­-Su­per­vised Learn­ing for Au­to­matic Speech Recog­ni­tion”, Zhang et al 2020 (pre­train­ing en­ables scal­ing to 1b-pa­ra­me­ter effi­cien­t-at­ten­tion Con­form­ers)

Genetics:

Every­thing Is Her­i­ta­ble:

• “Mul­ti­vari­ate ge­nomic analy­sis of 1.5 mil­lion peo­ple iden­ti­fies genes re­lated to ad­dic­tion, an­ti­so­cial be­hav­ior, and health”, Lin­ner et al 2020; “Hap­pi­ness and Well­be­ing; the value and find­ings from ge­netic stud­ies”, Van de Wei­jer et al 2020
• “Com­bined Util­ity of 25 Dis­ease and Risk Fac­tor Poly­genic Risk Scores for Strat­i­fy­ing Risk of Al­l-Cause Mor­tal­ity”, Meis­ner et al 2020
• “Ex­ome se­quenc­ing and char­ac­ter­i­za­tion of 49,960 in­di­vid­u­als in the UK Biobank”, Van Hout et al 2020 (“2% of this pop­u­la­tion has a med­ically ac­tion­able vari­ant”)
• “Ex­plor­ing the vari­ance in com­plex traits cap­tured by DNA methy­la­tion as­says”, Bat­tram et al 2020 (epigenome her­i­tabil­i­ty/­vari­ance-com­po­nents: ~0 across 400 traits)

En­gi­neer­ing:

• “The sec­ond decade of syn­thetic bi­ol­o­gy: 2010–2020”, Meng & El­lis 2020
• Em­manuelle Char­p­en­tier & Jen­nifer Doudna win No­bel Prize for CRISPR

Statistics/Meta-Science/Math

• “Pub­li­ca­tion rate in pre­clin­i­cal re­search: a plea for pre­reg­is­tra­tion”, van der Naald et al 2020 (me­dia; ~75% of sac­ri­ficed an­i­mals yield no pub­lished re­sult)
• “The Em­pir­i­cal Meta­math­e­mat­ics of Eu­clid and Be­yond”, Stephen Wol­fram (an­a­lyz­ing El­e­ments as a DAG); “Singing Eu­clid: the oral char­ac­ter of Greek geom­e­try”, Vik­tor Blåsjö (see also “Hav­ing Had No Pre­de­ces­sor to Im­i­tate, He Had No Suc­ces­sor Ca­pa­ble of Im­i­tat­ing Him”)

Politics/Religion

• Im­ma­nen­tiz­ing the Es­cha­ton, An­abap­tist edi­tion: Mün­ster re­bel­lion; the wreck of the Batavia

Psychology/Biology

• “Effects of caloric re­stric­tion on hu­man phys­i­o­log­i­cal, psy­cho­log­i­cal, and be­hav­ioral out­comes: high­lights from CALERIE phase 2”, Dor­ling et al 2020
• “Ques­tions About Trees”, Brian Hayes (“the para­dox of the plank­ton”: what main­tains di­ver­sity in forest­s?)
• “The hear­ing aid dilem­ma: am­pli­fi­ca­tion, com­pres­sion, and dis­tor­tion of the neural code”, Arm­strong et al 2020 (to­wards hear­ing aids de­signed by black­box op­ti­miza­tion to make neural pat­terns match pat­terns of nor­mal hear­ing?)
• “The role of cat eye nar­row­ing move­ments in cat-hu­man com­mu­ni­ca­tion”, Humphrey et al 2020
• En­cephali­tis lethar­gica (“A vi­ral in­fec­tion of the mind? The cu­ri­ous case of en­cephali­tis lethar­gica”)
• “Brain Train­ing Habits Are Not As­so­ci­ated With Gen­er­al­ized Ben­e­fits to Cog­ni­tion: An On­line Study of Over 1000 ‘Brain Train­ers’”, Sto­janoski et al 2020
• “C₆₀ in olive oil causes light–de­pen­dent tox­i­c­ity and does not ex­tend lifes­pan in mice”, Grohn et al 2020 (Back in 2012, I found the orig­i­nal re­sult un­be­liev­able for many rea­sons¹; but in case any­one still takes it se­ri­ous­ly…)

Technology

• Lib­Gen/S­ci-Hub full­text search en­gine: search 5.5m+ down­load­able book­s/77.5m+ pa­pers (great al­ter­na­tive to Google Book­s/Schol­ar)
• “Pro­fil­ing a ware­house­-s­cale com­puter”, Kanev et al 2015 (pre­vi­ous­ly: Ren et al 2010)
• “Ex­pert judg­ment on mark­ers to de­ter in­ad­ver­tent hu­man in­tru­sion into the Waste Iso­la­tion Pi­lot Plant”, Trauth et al 1993 (the San­dia long-term nu­clear waste warn­ing mes­sage ex­pert re­port for the WIP)
• “The Elu­sive Peril of Space Junk: Mil­lions of hu­man ar­ti­facts cir­cle the Earth. Can we clean them up be­fore they cause a dis­as­ter?” (the Kessler syn­drome never went away)
• “The rise and fall of Adobe Flash: Be­fore Flash Player sun­sets this De­cem­ber, we talk its legacy with those who built it”; “How Flash Games Shaped The Video Game In­dus­try: Flash is dead. But the in­flu­ence of Flash games on mod­ern game­play is in­escapable”, Jonas Rich­n­er; “Re­liv­ing Flash Game His­tory”, Alex Ir­pan (why were Flash browser games so cre­ative? The Flash scene ben­e­fited from easy en­try us­ing an in­te­grated pro­gram­ming en­vi­ron­ment, smal­l­-s­cale games en­abling rapid it­er­a­tion, many ex­em­plars to im­i­tate, envy and ad­mi­ra­tion dri­ving im­i­ta­tors, the con­vic­tion that im­prove­ments were pos­si­ble, a global tal­ent pool with lots of free time, and game web­sites like New­grounds to cu­rate games to cre­ate a so­phis­ti­cated & de­mand­ing user­base which could drive mega-traffic to hits, in a vir­tu­ous cy­cle of re­cruit­men­t/­com­pe­ti­tion/feed­back­/im­prove­ment. See also Bakewell.)
• “This page is a truly naked, Bru­tal­ist HTML quine.”; A self­-build­ing web­site

Fiction

• “My Dad, the Pornog­ra­pher” (on An­drew J. Offutt)
• The Mon­kees (from pass­ing ref­er­ences, I was al­ways con­fused if ‘The Mon­kees’ were a TV show or band—ap­par­ently they were both! Best-selling ones, too.)